Secure DevOps Kit for Azure

AzSK ARM Template Checker

This is a very useful open source tool used internally by Microsoft to validate that best practices are being followed in their Azure ARM templates.

This short post shows how to incorporate the AzSK ARM Template Checker into your Azure YAML Pipeline.

If you want to use a Linux build agent, you can use the PowerShell task to run AzSK.

- task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: |
              Set-PSRepository -Name "PSGallery" -InstallationPolicy Trusted
              Install-Module AzSK
              Import-Module AzSK
              Get-AzSKARMTemplateSecurityStatus -ARMTemplatePath $Env:BUILD_SOURCESDIRECTORY/arm-templates
            failOnStderr: true

Otherwise if you are using a Windows build agent, you can use the Azure Extension

January 12, 2020

Counting word frequency using map.merge

Imagine we want to produce a Map with each unique word as a key and it’s frequency as the value.

Prior to Java 8 we would have to do something like this:

Map<String, Integer> map = new HashMap<>();
for(String word : words) {
    if(map.containsKey(word)) {
        int count = map.get(word);
        map.put(word, ++count);
    } else {
        map.put(word, 1);
    }
}

However with the map.merge method we can now do:

Map<String, Integer> map = new HashMap<>();;
for(String word : words) {
    map.merge(word, 1, Integer::sum);
}
return map;

Pretty nice eh?

April 7, 2019January 6, 2021

Email is dead, long live Slack

Here are some great reasons to stop using email for team communication and instead switch to Slack

By default email messages are private – only available to the recipients. Slack messages are by default available to the whole team. Simply join the channel you’re interested in (or leave if not). How many times have you had to forward an email to someone who wasn’t on the original? or worse that other person never got to give their valuable contribution because they were never on the list?
Build a knowledge base – with email when someone leaves your company their account is deactivated and along with it all their sent emails. Imagine how useful this info could be if preserved and made searchable! Key decisions, how-tos and historical context can be available throughout the project and made available to all especially new comers.
Marketplace of apps – Slack has LOTS on fantastic integrations, like Git, Jenkins and JIRA which help to keep task communications flowing. Eg. see when a code review is required and openly discuss.
Self service – No need to request mailing lists from your email admin for topics or projects, simply create a channel and invite the relevant team members. E.g. Just developers working on project X
Multimedia – Call a video conference and screen share from within a shared channel without having to mess around with other conferencing apps. You can even give others control over your desktop (useful if it’s going to take too long to explain a technical task)
Sync and Async – Conversations are much closer to real-time than email, but still have the option of being asynchronous if you don’t want to be distracted.
Connection – Remote team mates feel more connected with Slack. You can see who’s online. You can see other work happening even if you’re not directly involved with the project or you can simply have a bit of banter with fellow employees easily without worrying about who should I CC in this email.
Strangers are friends – Other companies can be given access to a specific Slack channel and feel part of the team.
Don’t repeat yourself – No huge email chains with reply to all that require you to scroll through pages of crap to find the context on the conversation.

March 5, 2019

Spring Boot Microservice Integration Test Using Hoverfly

WebTestClient used to simplify external web service calls into microservice
SpringBootTest enabled with WebEnvironment defined port to enable webserver
Hoverfly to mock external webservice and provide precanned responses

package hello;

import io.specto.hoverfly.junit.core.HoverflyConfig;
import io.specto.hoverfly.junit.rule.HoverflyRule;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.test.web.reactive.server.WebTestClient;

import static io.specto.hoverfly.junit.core.SimulationSource.defaultPath;

@RunWith(SpringRunner.class)
@SpringBootTest(
		webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT,
		classes = hello.Application.class)
public class IntegrationTests {

	@ClassRule
	public static HoverflyRule hoverflyRule = HoverflyRule.inSimulationMode(defaultPath("sm9-create-ticket.json"),
			HoverflyConfig.localConfigs().asWebServer().proxyPort(8500));

	@Autowired
	private WebTestClient webClient;

	@Test
	public void testCreateTicket() {

		this.webClient.get().uri("/ticket?query=x").exchange().expectStatus().isOk()
				.expectBody(String.class).isEqualTo("Response from HPSM for create ticket");

	}

}

February 24, 2019

Continuous Integration, Continous Delivery, Continous Deployment.

Within DevOps the terms Continuous Integration, Continuous Delivery and Continuous Deployment get thrown around a lot. Here is the simplest definition I could come up with to quickly explain each to a non techie like a project manager.

Continuous Integration	Running unit and other tests on every branch on every commit and merging to master every day
Continuous Delivery	As above but each commit CAN be pushed to production
Continuous Deployment	As above but each commit IS pushed to production

February 24, 2019

Serverless Continuous Deployment for Java AWS Lamba using AWS CodePipeline

This post shows step by step how to deploy your serverless Java AWS Lambas continuously to production. Moving from pull request, merge, build, deploy and finally test.

Overview

Project Setup

For our project we are going to assume a standard Maven Java project structure, with Cloudformation and build specification config in the root of the project.

Within the Maven pom.xml file, you must include the lambda core libraries.

<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.2.0</version>
</dependency>
And also include the AWS SDK Java BOM
<!–https://aws.amazon.com/blogs/developer/managing-dependencies-with-aws-sdk-for-java-bill-of-materials-module-bom/–>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bom</artifactId>
<version>${com.amazonaws.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
Next you also need to ensure that the JAR artifact is built flat
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation=”com.github.edwgiz.mavenShadePlugin.log4j2CacheTransformer.PluginsCacheFileTransformer”>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>com.github.edwgiz</groupId>
<artifactId>maven-shade-plugin.log4j2-cachefile-transformer</artifactId>
<version>2.8.1</version>
</dependency>
</dependencies>
</plugin>
…

Build Pipeline using AWS CodePipeline

Source Step

The first step in the AWS CodePipeline is to fetch the source from the S3 bucket

Action Name: Source
Action Provider: S3
Bucket: <your release bucket>
S3 Object Key <path of your application>.zip
Output Artifact: MyApp

Build Step

Next step in the pipeline, you need to configure a CodeBuild project.

Set the current environment image to aws/codebuild/java:openjdk-8

Use the following buildspec.yml in the root of your project:

version: 0.2
phases:
build:
commands:
– echo Build started on `date`
# Unit tests, Code analysis and dependencies check (maven lifecycle phases: validate, compile, test, package, verify)
– mvn verify shade:shade -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=error
– mv target/MyApp-1.0.jar .
– unzip MyApp-1.0.jar
– rm -rf target tst src buildspec.yml pom.xml MyApp-1.0.jar
– aws cloudformation package –template-file main.yaml –output-template-file transformed_main.yaml –s3-bucket myapp-prod-outputbucket-xxxxxxxx
cache:
paths:
– ‘/root/.m2/**/*’
artifacts:
type: zip
files:
– transformed_main.yaml

Staging Step

After the artifact is built, we now want to create a change set using CloudFormation.

Action mode: Create or replace a change set
Template: MyAppBuildOut::transformed_main.yaml
Stackname: <name of your created stack here>

Define your Lambda function using Java (using serverless format), in your Cloudformation config file, placed in the root of your Maven project.

LambaFunctionName:
      Type: AWS::Serverless::Function
      Properties:
        Handler: au.com.nuamedia.camlinpayment.handler.MenuHandler
        Runtime: java8
        Timeout: 10
        MemorySize: 1024
        Events:
          GetEvent:
            Type: Api
            Properties:
              Path: /menu
              Method: get

Deploy Step

The ChangeSet can then be executed and the changes automatically rolled out to production safely. Any problems encountered and an automatic rollback occurs.

Action Mode: Execute changeset
Stackname: <name of your created stack here>
Change set name: <change set name from previous step>

Outcome

Congratulations! you now have your Java AWS Lamba functions deploying to production using Continuous Deployment. AWS CodePipeline is easily configurable via the UI and can also be defined as code and stored in version control.

December 31, 2018

Epics, User Story and Task, what’s the difference?

Type	Duration	Target	Scope	Estimate	Displayed on
Epic	Multiple sprints over a few months	Product Owner	High level objective	–	Backlog
User Story	Single sprint	End user or customer	Feature level	Story Points	Backlog
Task	Few days	Development Team	Assignable to individual	Hours	Agile board

April 29, 2018

Do your dependencies leave you open to attack?

According to the 2015 Verizon Data Breach Investigations Report (DBIR). 98% of attacks are opportunistic in nature, and aimed at easy targets. The report also found that more than 70% of attacks exploited known vulnerabilities that had patches available.

The recent breach at Equifax was caused by a known vulnerability in the popular Struts web framework library, when uploading files. It took Equifax at least two weeks after the attack to discover the data breach and this was almost four months after the exploit had been made public. Automated alerting on known exploits could have prevented this catastrophic security hole.

This post shows an automated way to check your third party library dependencies to ensure your site does not become a victim to these opportunistic attacks.

We will use the dependency checker provided by OWASP. This example shows integration with a Maven build where the check is run against every build during the verify stage. The first run will take a while as it has to download the entire vulnerability database. Subsequent runs will have this cached and so will run much faster.

Maven dependency include:

         <dependency>
            <groupId>org.owasp</groupId>
            <artifactId>dependency-check-maven</artifactId>
            <version>${org.owasp.dependency-check-maven.version}</version>
            <scope>test</scope>
        </dependency>

Maven plugin configuration:

             <plugin>
                <groupId>org.owasp</groupId>
                <artifactId>dependency-check-maven</artifactId>
                <version>3.1.2</version>
                <configuration>
                    <cveValidForHours>12</cveValidForHours>
                    <failBuildOnCVSS>8</failBuildOnCVSS>
                </configuration>
                <executions>
                    <execution>
                        <phase>verify</phase>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

Maven command to run:

mvn org.owasp:dependency-check-maven:check

Reference

https://www.triology.de/en/blog-entries/automatic-checks-for-vulnerabilities-in-java-project-dependencies

https://jeremylong.github.io/DependencyCheck/dependency-check-maven/index.html

https://www.owasp.org/index.php/OWASP_Dependency_Check

February 25, 2018

The Four Tendencies

Gretchen Rubin says a useful way to think about people’s behavior is by considering how willing they are to meet or resist expectations on them. Expectations can either be external, like your boss asking for a project to be completed or internal, like exercising regularly.

From these she identifies the four combinations labelled as The Four Tendencies.

	External Expectation	Internal Expectation
Upholder
Questioner
Obliger
Rebel

Meets Resists

This could provide you with more empathy when considering your colleagues, friends or family and make you a more effective communicator. Maybe try thinking about members of your software development team and which tendency they seem to exhibit.

Which one are you? Take the quiz and find out.

February 7, 2018

Achieving PCI compliance the easy way with a serverless architecture

Achieving PCI Compliance can be a rather onerous ongoing commitment. The first thing you will have to show is your architecture in the form of a network diagram together with a data flow diagram showing the routes where credit card data is transmitted. The more complex the architecture the more work involved in making it PCI compliant. So what can you do to mimise the amount of effort? Well your first thought should be around minimising the scope that falls under PCI compliance. This can be done by isolating only the components requried for payments and moving everything else outside of this scope.Once this is done you should then think about how to implement the system that will be in scope for compliance and this is what I want to talk about in this post.

The key to easier PCI compliance of YOUR system is to offload as much as possible to other providers and by other providers I mean Cloud hosting providers like AWS and GCP who’s platforms hare already attained PCI compliance.

In this example I’m going to use AWS. I’m going to compare two architectures, one deploying your application on EC2 instances and the other using Lambda as our serverless facilitator.

Lets take a simple application that takes credit card details from customers over the phone line and uses Paypal payment gateway to process the payment.

High level network diagram

If we ignore the telephony section of the data flow to keep things simple in this illustration then these are the architectures that we have to produce in each case

Traditional Architecture

Network diagram for traditional architecture

AWS Components used

VPC
public subnets
private subnets
Internet gateway
NAT gateway
Two Availability zones
Internet Gateway
Bastion host for SSH
Security groups
ACLs
Route tables
S3 buckets for Cloudwatch logs

Serverless Architecture

Network diagram for serverless architecture