Operations Infrastructure Month in Review #3
What’s this about?
Today, we’re open sourcing Conveyor, a fast build system for Docker images.
Since we started using Docker in production over 5 months ago, one of our biggest pain points has been slow build times for our Docker images. Initially, we built our images in our CI pipeline using CircleCI. While this was a pragmatic first choice, it presented a few problems:
To address these problems, we needed a system that was:
A Docker image is simply a set of layers that, when combined, produces a full root file system with some additional metadata. Let’s take a look at a trivial Dockerfile and how it generates layers:
FROM alpine:3.1 ADD file /tmp/file
We can build this, then run the
history command to inspect the layers of the resulting image.
$ docker build -t example . Sending build context to Docker daemon 2.56 kB Step 0 : FROM alpine:3.1 ---> 8dd8107abd2e Step 1 : ADD file /tmp/file ---> 505aac3df7e2 Removing intermediate container 11cfabbf60ea Successfully built 505aac3df7e2 $ docker history example IMAGE CREATED CREATED BY SIZE COMMENT 505aac3df7e2 34 hours ago /bin/sh -c #(nop) ADD file:efd116bc99eeadc9e9 0 B 8dd8107abd2e 2 weeks ago /bin/sh -c #(nop) ADD file:f302eb46ca6d55d4ba 5.041 MB
As you can see, this generated two layers; one layer copied from the
alpine:3.1 base image, and another layer which adds a file to
Let’s try building the image again:
$ docker build -t example . Sending build context to Docker daemon 2.56 kB Step 0 : FROM alpine:3.1 ---> 8dd8107abd2e Step 1 : ADD file /tmp/file ---> Using cache ---> 505aac3df7e2 Successfully built 505aac3df7e2
You’ll notice that the output has an extra line after Step 1:
---> Using cache. For
COPY steps within Dockerfiles, Docker will hash the contents of the file or directory and only re-execute the step if the contents were changed. If the contents are the same, Docker will use the existing cached layer.
This seems of little value with a trivial example that just adds a single file, so let’s look at how we can optimize our Dockerfile for ruby application that uses bundler.
A naive Dockerfile might look something like this:
FROM ruby RUN mkdir -p /usr/src/app WORKDIR /usr/src/app COPY . /usr/src/app RUN bundle install
While this will work fine, there’s a subtle problem here; if we change any source code, the
COPY step will no longer be able to use the existing cached layer, which also means that any subsequent steps will also need to be re-run since they depend on their parent layer. Even if we didn’t change our Gemfile, Docker is going to run a fresh
bundle install, which will take a long time.
To get around this, a better Dockerfile will look like this:
FROM ruby RUN mkdir -p /usr/src/app WORKDIR /usr/src/app COPY Gemfile /usr/src/app/ COPY Gemfile.lock /usr/src/app/ RUN bundle install COPY . /usr/src/app
We instead add the Gemfile first, then run bundle install before we add any other files. This means that even if we change source code, we’ll always use the existing cached layer for the
bundle install step (unless the Gemfile changes of course). We just reduced our build time from a couple of minutes to under a few seconds.
Not only is our build time faster, but also when we go to push the new image to the docker registry, since the Docker daemon only needs to send 1-2 new layers. Now, all of this optimization is moot unless our build system has a pre-existing layer cache, which is not the case with the current Docker build systems like Docker Hub, and Quay.
Conveyor is our solution to building Docker images fast. It does this using a few key design decisions:
In addition, Conveyor:
pushevents and Conveyor will create pretty commit statuses for the build:
Eventually we plan to have Conveyor sport a RESTfull api for triggering builds and returning build statuses, as well as streaming log output.
We’ve been using Conveyor to build all of our Docker images now for the past couple of months and have been very pleased with the reduction in build and deploy time. Repos that used to take 6-7 minutes to build are now built in 1-2 minutes, making it easy for developers to get their changes into production as quickly as possible. If you’re having Docker build time woes, give it a try and report any issues to https://github.com/remind101/conveyor/issues!
This article has to do with understanding user research from an engineering perspective. Whether you’re curious about the different types...