How to stay DRY doing Microservices

TL;DR

When using Jenkinsfile and Dockerfile with Microservices you are typically repeating the same boilerplate code over and over again. Initially this is not a problem but as the number of Microservices – and Git branches – start to increase it can become quite painful.

In order to stay DRY you can leverage the ONBUILD Dockerfile keyword in a custom base image and a global var to define a reusable Jenkins pipeline.

With this solution each Microservice has a Jenkinsfile similar to

… and a Dockerfile as simple as

See Dockerfile for an example on how the custom base image can look like and the project pom on how build both the base image and the Microservices using it!

Read the rest of the article if you are interested in the details or if the above does not make much sense.

Context

Let’s say you are using Microservices. Following best practices, each Microservice has its own source code repository. You start out with a few but this rapidly grows to a few dozen and with time you are managing a few hundred, perhaps more.

Each Microservice is built as a Docker image that defines a few labels, variables and some bootstrap code. A Dockerfile might look something like:

Having fully bought into the whole DevOps and automation concept you have CI/CD pipelines to build and deploy your Microservices.

Jenkins you use out of habit, because this is what you know or because it seems to be a very popular choice. Who really knows why, but Jenkins is what you are currently stuck with.

Jenkins is configured to automatically detect new Git repositories and manage pipelines for each Git branch. A Jenkinsfile in your Git repository defines the steps for building a Microservice and pushing code automatically triggers the build.

Thus the Git repository root contains two boilerplate files, one for Docker and one for Jenkins:

In most cases these files are almost identical apart from label and environment values. In order to facilitate for developers these they are automaticallly generated using a template mechanism of some sorts. Perhaps the nifty Docker Enterprise Desktop shipping as a part of Docker EE 3.0

The Problem

Requirements change or perhaps there is a bug but for whatever reason either the standard Jenkins pipeline and or Dockerfile change over time.

With a large number of Microservices that each has its own version of the Jenkinsfile and Dockerfile, multiple branches for handling the master line, development, features and bugs the relevance of the DRY principle starts to sink in and you realize you have a WET solution…

For every change that comes along you now have to check out the code make the same edit for all branches.

Cheery-picking and merging helps but still there is a lot of typing, committing and pushing going. This increases the risk of typos and other inconsistencies, in particular if done by multiple teams and developers.

Of course, nothing prevents you from automating changes like this, but it would throwing code at a sympton instead of resolving the root cause.

A Solution

A solution that addresses the WET root cause comes in two parts:

  • Use a custom base image in your Dockerfiles
  • Use pipeline DSL in your Jenkinsfiles

Using a Custom Base Image

A custom base image should contain all things common to your downstream Microservices, things like

  • standard image labels
  • log configuration files
  • common bootstrap shell scripts
  • runtime users

The custom base image in itself uses a more standard upstream image, such as the alpine version for your language.

Some things depend on the downstream Microservice, however. What if you have a label containing the name or version of the Microservice? These are only known at build time of the Microservice itself, not when we are building the base image.

The key here is Docker’s ONBUILD keyword which allows you to customize your base image to your Microservice using build time arguments.

For example we can set a version container label on the downstram Microservice with

Given that the base image is called mybase we then use a Dockerfile in Microservice repository with the following content:

That’s it. That is the entire Dockefile.

To build it we have to supply a build argument to the docker command, something along the lines of

The resulting image will then contain the container label version: 1.3.9 (surprise!).

If the we want to change something, perahps adding another standard label, update the logic to the bootstrap script, change the log configuration or perhaps just update the to the latest alpine for Java, we only change it in one place, i.e. in the custom base image mybase.

Of course, we still need to trigger the rebuild and redeployment of all our Microservices but the DRY principle is respected.

The beauty of this approach is that we still have our Dockerfile in each Git repository which can be customized if there is a real need. Just because there is a default base image does not mean that we force everybody to use it all the time. All our CI/CD tools will still work exactly the same and the only drawback is that we now have one exception to manage separately from the rest.

If you find yourself making a lot of similar exceptions, refactoring of the base image might be in place. Or perhaps there is a need for two different types of base images? In any case, try to Keep it as Simple and Stupid as possible.

Please see Dockerfile for a more complete example of how a custome base image can look like!

Use pipeline DSL in your Jenkinsfiles

Jenkins has support for building custom pipeline DSL in Groovy.

This can be quite complicated given that Jenkins Groovy flavour is not 100% vanilla and it has a sour twist – not all Groovy features are available and you need follow certain conventions. If you stick to existing steps and very simple groovy code you should be good though.

Here we will not go into details on how to develop a Pipeline DSL but the basic idea is that you define a global variable for the entire pipeline and use it your Jenkinsfiles.

Given a DSL library called mylib and a global var called mavenPipeline the Microservice Jenkinsfile could be as simple as this:

Note the trailing underscore which is a package placeholder to which the annotation is attached!

Here we assume that mavenPipeline has parameter support for the java version to use. You can have others as well, if you like, but be careful not to repeat too many boilerplate settings!

Now, if there is a change to how the Microservice is built, how continous delivery is done or perhaps if a new quality gate is added as a separate step, there is no need to change any of the Microservices. You just change the definition of the mavenPipeline.

Similar to how we handle the Dockerfile we can manage a custom Jenkinsfile if needed.

An Example

This Git repository contains a more complete example of a custom base image. It assumes Java based Microservices built by Maven but the concept should be easily adaptable to any language or platform.

To build the base image you can launch

If you want to deploy the base image you need to

  • Configure Maven distributionManagement settings; and
  • Make sure to change the docker.namespace to you Docker Hub account.

If you are using a private registry you also need to update the docker.registry property and probably add credentials in Maven’s settings.xml – YMMV!

Once Maven is properly configured you can then run…

… and Maven will happily deploy the (empty) JAR to your Maven registry and Docker image to Docker Hub or whatever you have configured.

Your Microservices should use Dockerfile along the lines of Dockerfile.microservice in the root of each Microservice repository branch.

(!) Note that the example requires Java 8 and Maven 3.5 to run!

Jenkinsfile

The Jenkinsfile will not work without you writing a custom DSL and adding the mavenPipeline global variable to your Jenkins instance.

Using a Jenkinsfile works best with a plugin like Bitbucket Branch Source Plugin or similar so that all new repositories and branchs are automagically discovered by Jenkins.

POM

The Maven POM builds the Docker image using Spotify’s dockerfile-maven-plugin.

The POM is configured to

  • Download all dependencies which are then added to the image
  • Create an executable JAR file which is also added to the image
  • Build the image if a Dockerfile is present using a Maven docker profile

The base image pom.xml has two dependencies: the logback-classic and logstash-logback-encoder libraries. These are shared by all our Microservices and are required for the logback.xml configuration file.

The configuration found in the POM can be used both for building the base image and for building the Microservices, indeed given the use of a Maven docker profile it can be used for any Maven project with or without Dockerfiles.

In order to stay DRY the plugin configuration in the POM should be put in a parent pom shared by all Microservice projects. Or you will again end up with a WET solution…

Entrypoint

The bootstrap.sh shell script that serves as the docker image entry point.

It sets a few variables used for configuring the Microsevice and launches the microservice iteslf with a number of standard JVM arguments, including configuration of logback and JMX.

A nifty feature is that it will pass along all docker command line arguments to the Java runtime.

Last Words

There are of course other solutions that still respect the DRY principle.

For example it might be preferable to get rid of the Dockerfile altogether. If you are building with Maven a good candidate is to use JIB.

You could also get rid of the Jenkinsfile and implement autodicsovery yourself but this seems like a rather cumbersome approach though. I would opt for an out-of-the-box solution that provisions build pipelines automatically.

I am not aware of any alternaives that do not use a Jenkinsfile, so it may be a good reason to look beyond Jenkins. USing another tool also has the benefit of not having to deal with Jenkins flavour of Groovy…

Any pointers on good alternatives that stay DRY for managing a large number of Git repositories and Dockerfiles are welcome!

Exception handling using enumerations in Java (II)

In the first part of this blog post I discussed how Java Enumerations can be conveniently used as

  1. Fault codes in exceptions; and
  2. Provide formatted and localized error messages

The major drawback with the approach is that the fault codes make exception handling more complicated for cases when the same logic should be applied to different enumeration types.

This part of the blog post discusses a possible extension of Java Syntax for better exception handling when using enumeration fault codes.

First we need a way to distinguish the exceptions that support fault code enumerations from those that do not. We also need to indicate what type of fault code is supported by the exception. We therefore introduce a a new Fault interface from which all fault code exceptions must inherit:

If we for a given application assume an ApplicationFault Exception class defined as

… we can then specify the fault code as constant values for the exception, i.e. something like

This new syntax should be interpreted as

  • Handle all ApplicationExceptions for fault codes Code.A1 and A2 and
    IOExceptions in the same way
  • Handle ApplicationExceptions for Code.A3 faults separately
  • Allow all ApplicationExceptions for Code.A4 faults to propagate up the
    call chain

(The Code enumeration type can be deduced from the ApplicationException.)

In addition, compile time errors would occur if

  • Non-existing or the wrong enumeration fault codes for the class are referenced
  • ApplicationFault does not implement Fault
  • ApplicationFault is a checked exception and not all fault codes are caught

We could even take it one step further if Java would allow generic type parameters for Exception classes. (Exceptions in Java do not support generic type parameters as there is no evident use for them, but if we introduce fault codes they would start making sense.)

Let us assume that the JDK defines a generic RuntimeFault:

We would then only need to define our fault codes in an enumeration (Localized or not), throw them as follows:

… and catch them like so

The main motivation for allowing generics in this way would be to have fewer exceptions. Instead of defining a new exception for each application or module with its own set of fault codes we would only define the fault codes and in most or all cases use the default RutimeFault exception class.

While this might be tempting we need to answer a number of questions introduced by the additional complication of allowing generics in exceptons:

  • Would other exceptions besides RuntimeFault be allowed to be generic in the
    same way?
  • Would the generic type parameters be limited to enums? If not, what would that
    mean?
  • Should we perhaps allowing catching the interface Fault as well? What class
    object would be assumed in the catch clause?

There are probably limitations related to type erasure and how the catch clause is implemented in Java that would answer these and other questions. Unfortunately I lack the insight to provide a good answer on the feasibility of any of the extensions suggested above.

In a blog post from last year the author proposes changes to Java 9 that pretty much attempts to solve the same problems as discussed in this post, but instead of using enum codes the solution is based on Strings. He actually goes as far as making a forked JDK to test the the new feature!

To catch the codes the author introduces a new syntax:

The CodedException class is required to be part of the catch clause for the codes to be caught.

To me the syntax is a bit clumsy as the CodedException is not kept together with the codes and using String constants is error prone. How do we known the fault code is ever thrown? With enumerations you get type safety and all codes has at least to be defined somewhere.

Last defining a single code is a bit more verbose, compare:

… as opposed to

Granted, there is a little bit of overhead not shown where we store the MessageFormat in the enumeration but this is quickly regained when there are more than a few codes.

Exception handling using enumerations in Java (I)

When I first read about Swift I quite liked its error handling.

In Java checked exceptions have gotten quite a bad rep but instead of throwing out the baby with the bathwater and getting rid of checked exceptions altogether, the way they do it in Swift seems to strike a good balance.

The thing that caught my eye however, was the use of enumerations instead of exception classes.
Continue reading

What is Strategy?

As an assignment in the Strategy and Marketing course of my ongoing MBA studies we were required to give our personal view of what strategy is and I thought I should publish the result here. Part of the reading for the assignment included five articles from Harvard Business Review, three by Michel Porter and two by Kathleen Eisenhardt.

The three articles by Porter takes a high level perspective, defining strategy in terms of the nation state, the industry as a whole and strategy at the highest level of the enterprise. Essentially this is a top-down perspective, or perhaps one could call it outside-in. For Porter strategy is about making conscious trade-offs (what not to do), supported by a unique set of activities that reinforce each other and are difficult to reproduce, all in all creating a unique strategic position giving a sustainable competitive advantage (What is Strategy, p 70).

In the two articles by Eisenhardt et al strategy is something emergent, arising from the pattern of activities (patching and the application of simple rules). It is more about the internal strategy process than finding a strategic position. Essentially this is more of a bottoms-up perspective on strategy, or perhaps one could say inside-out.

Porter and Eisenhardt also differ in the time scale of strategy. Strategy in the eyes of Porter is long term; if a strategy does not lead to a sustainable advantage with a high performance outcome, it is not strategy or at least not successful strategy. As a counter point, the two articles by Eisenhardt et al discuss strategy with a shorter horizon (“internet time”).

Continue reading

Diaspora Seeds

Recently a couple of New York college students made a stir when they announced that they would write an open source and distributed social network server as an alternative to Facebook. This is really in the true spirit of the internet, where everyone can set up his or her own server and have them commicate, compare e.g. how email works. Incidentally, this is the way Instant Messaging should work but only SIP and Jabber (used by e.g. Google Talk) actually implement. Evidently they have raised about $170 000 in three weeks using Kickstarter, their initial target was $10 000.

I sympathize 100% with their stated goals and will make sure I set up a Diaspora Seed once they make their first release.

More information on the Diaspora blog.

Toyota and Waterfall methdology for developing software

As you may know Agile software development is inspired by the principles of Lean Manufacturing which in turn is derived from the Toyota Production System (TPS). One Agile methodology, Lean Software Development even borrows the name Lean to make the connection explicit. Since the principles in Lean and TPS have much in common with how Japanese always have approached manufacturing and that TPS supposedly permeates the whole of Toyota, it comes as a real surprise that Toyota uses a waterfall process for software development!

REST for Scott

At the thanksgiving dinner of a friend from work I tried to explain REST-ful applications – and I think I failed miserably. Even though there are plenty of good introductions out there I felt I needed to organize my thoughts a bit. Having some spare time on the train on the way home I made a second, more structured attempt in the form of this blog post.

Imagine you are a web developer and is asked by a friend with a travel agency to help out create a web site for his company. Among the services on the site is a small web application for searching for hotels recommended by the agency. These hotels have been carefully vetted for the agency’s customer segment and useful hotel information is stored in a database.

Continue reading

SOA is Dead (but the corpse is still kicking)

In a famous post from January this year (which I missed) Ann Thomas Manes declares SOA dead. She is a Reasearch Director within the Burton Group, has worked with SOA in the Burton Group and elsewhere, and is a co-author of the WS-* specifications.

The post has sparked fierce defense from vendors like Oracle. During the half day SOA Architect forum in Geneva this Tuesday, Oracle spent the first seminar repeating that “SOA is not dead” like a mantra, I guess in order to reinforce the message. It seemed to be one of the mayor points they wanted to get across.

Continue reading

Universal Assembly Cache – Dependency Management for Mono and Microsoft.NET

In the Java world there are several ways of handling dependencies between project components in a shared development environment. Maven is a generic build tool that creates a standardized way of creating, building and publishing component artifacts. Dependencies between the different components are declared in a Maven build file and Maven automatically downloads the dependencies and all transient dependencies recursively from local or public repositories. Repositories can be local to a particular developer, department or enterprise; or they can be public, available to the development community at large. Ivy is another utility which focus solely on dependency management. It is positioned as the lean and simple alternative to Maven which focus only on the dependency aspects and that integrates well with Ant (the generic java build tool).

For .NET based solutions there does not seem to be neither a Maven nor Ivy style dependency management system for development. This article outlines a solution to dependency management in the .NET world that has been inspired by the Maven and Ivy.

Continue reading