Introducing Conveyor: A fast build system for Docker images

Today, we’re open sourcing Conveyor, a fast build system for Docker images.

Since we started using Docker in production over 5 months ago, one of our biggest pain points has been slow build times for our Docker images. Initially, we built our images in our CI pipeline using CircleCI. While this was a pragmatic first choice, it presented a few problems:

1. When building in CI (or via automated builds in Docker Hub), the build starts fresh without any cached layers. This made build times around 6-7 minutes on average, sometimes pushing 10 minutes for Dockerfiles with expensive build steps. Initially we tried pulling the last built image for the branch we were on, but a number of pre Docker 1.8 bugs bit us (for the record, I’m now a happy panda).
2. If the build got queued by CI, this would block being able to deploy the Docker image to Empire.

To address these problems, we needed a system that was:

1. Optimized to use cached layers, which is necessary to build images quickly.
2. Could be run on our own dedicated infrastructure to avoid queueing or noisy neighbors slowing down our build times.

Docker layer cache explained

A Docker image is simply a set of layers that, when combined, produces a full root file system with some additional metadata. Let’s take a look at a trivial Dockerfile and how it generates layers:

FROM alpine:3.1
ADD file /tmp/file

We can build this, then run the history command to inspect the layers of the resulting image.

$ docker build -t example .
Sending build context to Docker daemon  2.56 kB
Step 0 : FROM alpine:3.1
 ---> 8dd8107abd2e
Step 1 : ADD file /tmp/file
 ---> 505aac3df7e2
Removing intermediate container 11cfabbf60ea
Successfully built 505aac3df7e2
$ docker history example
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
505aac3df7e2        34 hours ago        /bin/sh -c #(nop) ADD file:efd116bc99eeadc9e9   0 B
8dd8107abd2e        2 weeks ago         /bin/sh -c #(nop) ADD file:f302eb46ca6d55d4ba   5.041 MB

As you can see, this generated two layers; one layer copied from the alpine:3.1 base image, and another layer which adds a file to /tmp/file.

Let’s try building the image again:

$ docker build -t example .
Sending build context to Docker daemon  2.56 kB
Step 0 : FROM alpine:3.1
 ---> 8dd8107abd2e
Step 1 : ADD file /tmp/file
 ---> Using cache
 ---> 505aac3df7e2
Successfully built 505aac3df7e2

You’ll notice that the output has an extra line after Step 1: ---> Using cache. For ADD and COPY steps within Dockerfiles, Docker will hash the contents of the file or directory and only re-execute the step if the contents were changed. If the contents are the same, Docker will use the existing cached layer.

This seems of little value with a trivial example that just adds a single file, so let’s look at how we can optimize our Dockerfile for ruby application that uses bundler.

A naive Dockerfile might look something like this:

FROM ruby
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . /usr/src/app
RUN bundle install

While this will work fine, there’s a subtle problem here; if we change any source code, the COPY step will no longer be able to use the existing cached layer, which also means that any subsequent steps will also need to be re-run since they depend on their parent layer. Even if we didn’t change our Gemfile, Docker is going to run a fresh bundle install, which will take a long time.

To get around this, a better Dockerfile will look like this:

FROM ruby
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY Gemfile /usr/src/app/
COPY Gemfile.lock /usr/src/app/
RUN bundle install
COPY . /usr/src/app

We instead add the Gemfile first, then run bundle install before we add any other files. This means that even if we change source code, we’ll always use the existing cached layer for the bundle install step (unless the Gemfile changes of course). We just reduced our build time from a couple of minutes to under a few seconds.

Not only is our build time faster, but also when we go to push the new image to the docker registry, since the Docker daemon only needs to send 1-2 new layers. Now, all of this optimization is moot unless our build system has a pre-existing layer cache, which is not the case with the current Docker build systems like Docker Hub, and Quay.

Introducing Conveyor

Conveyor is our solution to building Docker images fast. It does this using a few key design decisions:

1. To maximize the number of cached layers that can be used, Conveyor will first pull the last built image for the git branch before building.
2. It uses Docker 1.8.1 to perform the build and push, which contains performance improvements for all of the issues mentioned in https://github.com/remind101/docker-build/pull/3#issuecomment-119719331.

In addition, Conveyor:

1. Runs the build within a Docker container itself, making it easy to scale out using something like Docker Swarm. The Docker image used to perform the build can also be configured if you need to customize how you build your Docker images. The default builder builds the image using Docker 1.8.1 and tags the resulting image with the git branch and git commit sha before pushing.
2. Integrates with GitHub. Builds are triggered via push events and Conveyor will create pretty commit statuses for the build:

Eventually we plan to have Conveyor sport a RESTfull api for triggering builds and returning build statuses, as well as streaming log output.

Give it a try

We’ve been using Conveyor to build all of our Docker images now for the past couple of months and have been very pleased with the reduction in build and deploy time. Repos that used to take 6-7 minutes to build are now built in 1-2 minutes, making it easy for developers to get their changes into production as quickly as possible. If you’re having Docker build time woes, give it a try and report any issues to https://github.com/remind101/conveyor/issues!