This post is also avaialble on GitHub together with all the source files!

When using Jenkinsfile and Dockerfile with Microservices you are typically repeating the same boilerplate code over and over again. Initially this is not a problem but as the number of Microservices - and Git branches - start to increase it can become quite painful.

In order to stay DRY you can leverage the ONBUILD Dockerfile keyword in a custom base image and a global var to define a reusable Jenkins pipeline.

With this solution each Microservice has a Jenkinsfile similar to

@Library("mylib@latest") _
mavenPipeline(java: '8')

... and a Dockerfile as simple as

FROM mybase:latest

See Dockerfile for an example on how the custom base image can look like and the project pom on how build both the base image and the Microservices using it!

Read the rest of the article if you are interested in the details or if the above does not make much sense.

• Context
• The Problem
• A Solution
• An Example
• Last Words

Context

Let's say you are using Microservices. Following best practices, each Microservice has its own source code repository. You start out with a few but this rapidly grows to a few dozen and with time you are managing a few hundred, perhaps more.

Each Microservice is built as a Docker image that defines a few labels, variables and some bootstrap code. A Dockerfile might look something like:

FROM openjdk:8-jdk-alpine

ENV SVC_HOME=/opt/microservice
ENV SVC_ETC=${SVC_HOME}/etc SVC_LIB=${SVC_HOME}/lib SVC_CACHE=/var/cache/microservice BOOTSTRAP=${SVC_HOME}/bootstrap.sh

RUN addgroup --system microservice \
    && adduser --system --ingroup microservice --home $SVC_HOME microservice

ADD src/files ${SVC_ETC}/
ADD bootstrap.sh ${BOOTSTRAP}

RUN mkdir -p $SVC_LOG $SVC_CACHE $SVC_ETC \
    && chown -R microservice:microservice $SVC_LOG $SVC_CACHE $SVC_HOME \
    && chmod +x ${BOOTSTRAP}

USER microservice

ARG JAR_FILE

ARG ARTIFACT_ID
ARG GROUP_ID
ARG VERSION

LABEL org.label-schema.vendor="My Company" \
    org.label-schema.name=$ARTIFACT_ID \
    org.label-schema.description="Yet another Microservice" \
    org.label-schema.usage="/README.md" \
    org.label-schema.version=$VERSION \
    org.label-schema.schema-version="1.0" \
    java.version=${JAVA_VERSION} \
    java.alpine.version=${JAVA_ALPINE_VERSION}

ENV \
    ARTIFACT_ID=${ARTIFACT_ID} \
    GROUP_ID=${GROUP_ID} \
    VERSION=${VERSION} \
    LOG_LEVEL=INFO \
    CACHE_DIR=${SVC_CACHE}/${ARTIFACT_ID} \
    ETC_DIR=${SVC_ETC}/${ARTIFACT_ID} \
    MICROSERVICE=${SVC_HOME}/${JAR_FILE}

ADD target/lib ${SVC_LIB}/
ADD target/${JAR_FILE} ${MICROSERVICE}

# HTTP
EXPOSE 9000
# JMX
EXPOSE 8181

ENTRYPOINT ["/opt/microservice/bootstrap.sh"]

Having fully bought into the whole DevOps and automation concept you have CI/CD pipelines to build and deploy your Microservices.

Jenkins you use out of habit, because this is what you know or because it seems to be a very popular choice. Who really knows why, but Jenkins is what you are currently stuck with.

Jenkins is configured to automatically detect new Git repositories and manage pipelines for each Git branch. A Jenkinsfile in your Git repository defines the steps for building a Microservice and pushing code automatically triggers the build.

pipeline {
    agent docker
    stage ('Build') {
        steps {
            ...
        }
    }
    stage ('Test') {
        steps {
            ...
        }
    }
    stage ('Verify') {
        steps {
            ...
        }
    }
    stage ('Publish') {
        steps {
            ...
        }
    }
}

Thus the Git repository root contains two boilerplate files, one for Docker and one for Jenkins:

Dockerfile
Jenkinsfile

In most cases these files are almost identical apart from label and environment values. In order to facilitate for developers these they are automatically generated using a template mechanism of some sorts. Perhaps the nifty Docker Enterprise Desktop shipping as a part of Docker EE 3.0

The Problem

Requirements change or perhaps there is a bug but for whatever reason either the standard Jenkins pipeline and or Dockerfile change over time.

With a large number of Microservices that each has its own version of the Jenkinsfile and Dockerfile, multiple branches for handling the master line, development, features and bugs the relevance of the DRY principle starts to sink in and you realize you have a WET solution...

For every change that comes along you now have to check out the code make the same edit for all branches.

Cheery-picking and merging helps but still there is a lot of typing, committing and pushing going on. This increases the risk of typos and other inconsistencies, in particular if done by multiple teams and developers.

Of course, nothing prevents you from automating changes like this, but it would throwing code at a symptom instead of resolving the root cause.

A Solution

A solution that addresses the WET root cause comes in two parts:

• Use a custom base image in your Dockerfiles
• Use pipeline DSL in your Jenkinsfiles

Using a Custom Base Image

A custom base image should contain all things common to your downstream Microservices, things like

• standard image labels
• log configuration files
• common bootstrap shell scripts
• runtime users
• ...

The custom base image in itself uses a more standard upstream image, such as the alpine version for your language.

Some things depend on the downstream Microservice, however. What if you have a label containing the name or version of the Microservice? These are only known at build time of the Microservice itself, not when we are building the base image.

The key here is Docker's ONBUILD keyword which allows you to customize your base image to your Microservice using build time arguments.

For example we can set a version container label on the downstream Microservice with

ONBUILD ARG VERSION
ONBUILD LABEL version=${VERSION}

Given that the base image is called mybase we then use a Dockerfile in Microservice repository with the following content:

FROM mybase:latest

That's it. That is the entire Dockefile.

To build it we have to supply a build argument to the docker command, something along the lines of

docker build --arg VERSION=1.3.9 .

The resulting image will then contain the container label version: 1.3.9 (surprise!).

If the we want to change something, perhaps adding another standard label, update the logic to the bootstrap script, change the log configuration or perhaps just update the to the latest alpine for Java, we only change it in one place, i.e. in the custom base image mybase.

Of course, we still need to trigger the rebuild and redeployment of all our Microservices but the DRY principle is respected.

The beauty of this approach is that we still have our Dockerfile in each Git repository which can be customized if there is a real need. Just because there is a default base image does not mean that we force everybody to use it all the time. All our CI/CD tools will still work exactly the same and the only drawback is that we now have one exception to manage separately from the rest.

If you find yourself making a lot of similar exceptions, refactoring of the base image might be in place. Or perhaps there is a need for two different types of base images? In any case, try to keep it as simple and stupid as possible (cf. KISS).

Please see Dockerfile for a more complete example of how a custom base image can look like!

Use pipeline DSL in your Jenkinsfiles

Jenkins has support for building custom pipeline DSL in Groovy.

This can be quite complicated given that Jenkins Groovy flavor is not 100% vanilla and it has a sour twist - not all Groovy features are available and you need follow certain conventions. If you stick to existing steps and very simple groovy code you should be good though.

Here we will not go into details on how to develop a Pipeline DSL but the basic idea is that you define a global variable for the entire pipeline and use it in your Jenkinsfiles.

Given a DSL library called mylib and a global var called mavenPipeline the Microservice Jenkinsfile could be as simple as this:

@Library("mylib@latest") _
mavenPipeline(java: '8')

Note the trailing underscore which is a package placeholder to which the annotation is attached!

Here we assume that mavenPipeline has parameter support for the java version to use. You can have others as well, if you like, but be careful not to repeat too many boilerplate settings!

Now, if there is a change to how the Microservice is built, how continuous delivery is done or perhaps if a new quality gate is added as a separate step, there is no need to change any of the Microservices. You just change the definition of the mavenPipeline.

Similar to how we handle the Dockerfile we can manage a custom Jenkinsfile if needed.

An Example

This Git repository contains a more complete example of a custom base image. It assumes Java based Microservices built by Maven but the concept should be easily adaptable to any language or platform.

• pom.xml for building the image
• bootstrap.sh docker entrypoint for launching the microservice
• jmxremote.access for configuring remote Java JMX access
• logback.xml for log configuration

To build the base image you can launch

mvn clean package

If you want to deploy the base image you need to

• Configure Maven distributionManagement settings; and
• Make sure to change the docker.namespace to you Docker Hub account.

If you are using a private registry you also need to update the docker.registry property and probably add credentials in Maven's settings.xml - YMMV!

Once Maven is properly configured you can then run...

mvn clean deploy

... and Maven will happily deploy the (empty) JAR to your Maven registry and Docker image to Docker Hub or whatever you have configured.

Your Microservices should use Dockerfile along the lines of Dockerfile.microservice in the root of each Microservice repository branch.

(!) Note that the example requires Java 8 and Maven 3.5 to run!

Jenkinsfile

The Jenkinsfile will not work without you writing a custom DSL and adding the mavenPipeline global variable to your Jenkins instance.

Using a Jenkinsfile works best with a plugin like Bitbucket Branch Source Plugin or similar so that all new repositories and branches are automagically discovered by Jenkins.

POM

The Maven POM builds the Docker image using Spotify's dockerfile-maven-plugin.

The POM is configured to

• Download all dependencies which are then added to the image
• Create an executable JAR file which is also added to the image
• Build the image if a Dockerfile is present using a Maven docker profile

The base image pom.xml has two dependencies: the logback-classic and logstash-logback-encoder libraries. These are shared by all our Microservices and are required for the logback.xml configuration file.

The configuration found in the POM can be used both for building the base image and for building the Microservices, indeed given the use of a Maven docker profile it can be used for any Maven project with or without Dockerfiles.

In order to stay DRY the plugin configuration in the POM should be put in a parent pom shared by all Microservice projects. Or you will again end up with a WET solution...

Entrypoint

The bootstrap.sh shell script that serves as the docker image entry point.

It sets a few variables used for configuring the Microsevice and launches the microservice itself with a number of standard JVM arguments, including configuration of logback and JMX.

A nifty feature is that it will pass along all docker command line arguments to the Java runtime.

Last Words

There are of course other solutions that still respect the DRY principle.

For example it might be preferable to get rid of the Dockerfile altogether. If you are building with Maven a good candidate is to use JIB.

You could also get rid of the Jenkinsfile and implement auto discovery for Jenkins pipelines yourself but this seems like a rather cumbersome approach though. I would opt for an out-of-the-box solution that provisions build pipelines automatically.

I am not aware of any alternatives that do not use a Jenkinsfile, so it may be a good reason to look beyond Jenkins. Using another tool also has the benefit of not having to deal with Jenkins flavor of Groovy...

Any pointers on good alternatives that stay DRY for managing a large number of Git repositories and Dockerfiles are welcome!

Recently a couple of New York college students made a stir when they announced that they would write an open source and distributed social network server as an alternative to Facebook. This is really in the true spirit of the internet, where everyone can set up his or her own server and have them commicate, compare e.g. how email works. Incidentally, this is the way Instant Messaging should work but only SIP and Jabber (used by e.g. Google Talk) actually implement. Evidently they have raised about $170 000 in three weeks using Kickstarter, their initial target was $10 000.

I sympathize 100% with their stated goals and will make sure I set up a Diaspora Seed once they make their first release.

More information on the Diaspora blog.

At the thanksgiving dinner of a friend from work I tried to explain REST-ful applications – and I think I failed miserably. Even though there are plenty of good introductions out there I felt I needed to organize my thoughts a bit. Having some spare time on the train on the way home I made a second, more structured attempt in the form of this blog post.

Imagine you are a web developer and is asked by a friend with a travel agency to help out create a web site for his company. Among the services on the site is a small web application for searching for hotels recommended by the agency. These hotels have been carefully vetted for the agency’s customer segment and useful hotel information is stored in a database.

Let us assume the web application manages the namespace under the URL

http://qualitytravel.com/hotel

One way to construct the application is by storing e.g. navigation state in a server side session or as post parameters in the HTTP requests. Or maybe use an AJAX user interface where parts of the page is updated asynchronously in the background. These approaches are, however, incompatible with the basic requirement of you friend the the travel agent, who frequently copies the URL of a particular hotel and sends it per email to potential customers. He would also like his customers and all other users of the web site to be able to send URLs, create bookmarks etc. Only by creating stable, re-usable URL’s is this easily accomplished. Thus you decide on a basic categorization scheme where each hotel is listed under

http://qualitytravel.com/hotel///

Where is the two letter ISO code for the country, the city name in english using only ascii characters and is a unique name of the hotel (unique at least within the country). By typing in a URL like

http://qualitytravel.com/hotel/se/stockholm/grand-hotel

… in a web browser you would end up with the HTML page describing the Grand Hotel in Stockholm, Sweden.

Your friend is very ambitious but would like to start out small. Uptime is of course important but not critical and you offer to host the site on your server at your place for a minor fee per month. (This would help pay the electricity and internet connection bills.)

Now, your friend has a business partner who organizes conferences in different countries around the globe. The business partner would like to link to specific hotels on your site, hotels your friend has arranged special deals and have pre-booked a number of rooms for. Since you are using stable and predictable URLs this is no big deal, for any particular conference the business partner can easily add the reference on a web page, add them in emails sent out to the participants and even in printed hard copy.

As the partnership between your friend and his business partner deepens, the business partner would like tighter integration between the two web sites. In particular he would like to create a mashup application where each conference has a map with all the hotels printed at the correct geographic location. Since your friend has stored the exact geo location of each hotel in his database it should not be too difficult to achieve.

Being well versed in XML and enterprise standards you now ponder a bit on web services. You could of course create a web service with a method which would accept a list of hotel identifiers and return a list coordinates… After some deliberation you come to the conclusion it would be easiest to use the URLs as identifiers. They are unambiguous and you do not have to expose the primary keys of the database to the internet and there is no need to create new unique external identifiers. On top of that adding the identifiers to the mashup will be easy since the URLs are already used on the conference web page.

Thinking a bit further you realize that in the future there potentially is  more information which could be useful, not just the coordinates. Right now you do not know exactly what information that would be, but if you return a list of XML documents containing basic information on each hotel and with geo-coordinates, you can always expend the XML document format to include additional information at a later stage. At least as long you are careful to keep the format backwards compatible. This you do by not changing the meaning and syntax of the existing XML and make sure the XML elements have the same position relative to the root element (i.e. that the XPath for the elements is stable).

Finally, you realize that instead of taking the pain of having to create a web service with SOAP, WSDLs etc you decide that each hotel URL should return the corresponding XML if the client asks to get the hotel information in XML format instead of HTML. This can easily be accomplished by using the standard HTTP content negotiation, i.e. the client sends a list of desired formats and the server responds with a document in one of the formats if it can.

The changes to the existing application are relatively minor, you only need to implement the content negotiation feature and in addition to render HTML you will need a mechanism to render XML. (In Java, using servlets, servlet filters and a templateing mechanism sounds like a good approach.)

The result is a huge success. Business is booming for your friend and traffic to your server is increasing at an alarming rate. You are sure that any minute your ISP will knock on the door telling you you are violating the fair use clause of your contract. In addition, uptime is becoming more of a concern. If not for the load so at least for redundancy you need two servers. You decide to move the web site to a hosting facility. Since state is not kept in a server side session you realize that clustering will be trivial. Each request is independent of the previous meaning it can land on either server without complication. This also means adding more servers to handle a larger load will be a breeze.

Some time later your friend has found an additional business partner – this guy for some reason wants to create an AJAX client which uses the hotel information. The AJAX guys, programming in client side Java script, do not really like XML and prefer something called JSON. Would it be possible to add JSON support? In addition it would be nice if you could create a downloadable PDF for each hotel…

Having the infrastructure already set up, all is needed is to add two new rendering templates, one for JSON and one for PDF’s and add the two new formats to the lists served by the server.

–

*The above scenario describes using REST – REpresentational State Transfer – for building a web application. The text strives to illustrate the fundamental REST-ful principle of decoupling the reference of an object from its actual representation. It can also be said that in REST you take a resource or document oriented approach to software development, i.e. development is data-driven. This is opposed to service orientation with an RPC style of defining interfaces where focus is not on the data but rather on what you do with data; the focus is on the verbs instead of the nouns. In REST the actions are a well defined minimum like the four most well known HTTP verbs: GET, POST, PUT and DELETE. Above, only the implied use of GET is shown.
*

Using REST principles is not constrained to web applications. It has been suggested that REST should be used instead of SOAP based web services in a SOA architecture, thus imbuing some of the webs scalability and robustness in an Enterprise Integraion scenario. Gartner has termed it WOA for Web Oriented Architecture. WOA can be seen as a more constrained version of SOA.

It has even been suggested that REST is useful within applications. How this can be done is described in four articles on The Server Side. Beware though, although well written they end up being a plug for NetKernel, a REST-ful application server built on something they call ROC – Resource Oriented Computing. I would say the jury is still out on question whether using REST internally in an application is a good alternative or not.

References

Wikipedia has a well written article on REST:

http://en.wikipedia.org/wiki/REST

Web Oriented Architecture

*http://hinchcliffe.org/archive/2008/02/27/16617.aspx
*

*The Server Side, A Restful Core:
*

http://www.theserverside.com/tt/articles/article.tss?l=ARESTfulCorePart1

http://www.theserverside.com/tt/articles/article.tss?l=ARESTfulCorePart2

http://www.theserverside.com/tt/articles/article.tss?l=ARESTfulCorePart3
http://www.theserverside.com/tt/articles/article.tss?l=ARESTfulCorePart4

In a famous post from January this year (which I missed) Ann Thomas Manes declares SOA dead. She is a Reasearch Director within the Burton Group, has worked with SOA in the Burton Group and elsewhere, and is a co-author of the WS-* specifications.

The post has sparked fierce defense from vendors like Oracle. During the half day SOA Architect forum in Geneva this Tuesday, Oracle spent the first seminar repeating that “SOA is not dead” like a mantra, I guess in order to reinforce the message. It seemed to be one of the mayor points they wanted to get across.

Excerpt from Ann Thomas Manes post SOA is Dead; Long Live Services

Successful SOA (i.e., application re-architecture) requires disruption to the status quo. SOA is not simply a matter of deploying new technology and building service interfaces to existing applications; it requires redesign of the application portfolio. And it requires a massive shift in the way IT operates. The small select group of organizations that has seen spectacular gains from SOA did so by treating it as an agent of transformation. In each of these success stories, SOA was just one aspect of the transformation effort. And here’s the secret to success: SOA needs to be part of something bigger. If it isn’t, then you need to ask yourself why you’ve been doing it.

The latest shiny new technology will not make things better. Incremental integration projects will not lead to significantly reduced costs and increased agility. If you want spectacular gains, then you need to make a spectacular commitment to change. Like Bechtel. It’s interesting that the Bechtel story doesn’t even use the term “SOA”—it just talks about services.

And that’s where we need to concentrate from this point forward: Services.

This falls in line with my understanding that SOA without BPR/BPM/Transformation really does not amount to much. You need the entire organization to support a process and service view on business and IT in order to see real benefit.

In another post SOA is Dead? It’s About Time by Kurt Cagle acknowledges Ann Thomas Manes view but complements the picture with adding that

Perhaps my biggest reservation about SOA had to be the fact that, at the end of the day, it was still an RPC model that concentrated primarily on calling APIs that differed from one provider to the next. The result of this thinking is the sea of APIs, where there are now tens of thousands of APIs, each of which doing things a little (or in some cases, a lot) differently from one another, with very little cohesion, and with little thought to the semantic complexity that comes when you have that many microlanguages all competing for programmer attention.

Purists may argue that over time the SOA model (especially the SOAP/WSDL model) had been moving towards a more messaging-oriented architecture, but I’d counter that all that a messaging queue does is to decouple the receipt of the message from the response – if the message processor invokes a service, it is still an RPC, especially when transactions are involved.

This is one of the reasons that I think that resource oriented services – RESTful services – are beginning to gain real traction even as the big-box SOA projects are falling to the accountant’s axe. The publish/subscribe model in which what you’re publishing are not blogs but data documents (think XBRL or HL7) performs the same type of decoupling that message-oriented SOA did, but completely abstracts the intent from the process of communication.

This is consistent with my view that SOA, which if done correctly, should be document or data driven and should rely on messaging. As an alternative to an ESB used mainly for RPC over SOAP Web Services, I think that an REST-ful approach should be seriously considered when attempting SOA. Some thinkers at Gartner have started talking about a WOA – Web Oriented Architecture – as further constrained subset of the SOA approach based on REST principles.

I believe the change will be heavily resisted by companies like Oracle and IBM since there is less money to be made on the simpler and less tool/platform-intensive REST-ful approach to enterprise computing. Also they all have invested heavily in the WS-* and RPC-style SOA corner and thus are reluctant to change. But change will come sooner or later and the ongoing economical crisis will be a strong driver.