Operations Infrastructure Month in Review #3
What’s this about?
Today, we are open sourcing our log streaming solution for Empire.
Since day one, we’ve been able to stream our logs to analytics and search platforms such as Sumologic, but we were missing a simple, immediate solution for real-time inspection of logs, something as simple as
emp log -a <app>, a la Heroku.
Heroku’s solution, Logplex is open source, but is difficult to setup and operationally complex - two things we try to avoid in Empire.
From the stream names, to the tags and the partition keys, everything is configurable via environment variables and go templates. You can use it however you want.
Since we are building 12-factor apps at Remind, we have always been logging to stdout, and logspout has been working extremely well for us. It attaches to all containers on a host, then routes their logs wherever we want. We use it to send our logs to Sumologic, and now thanks to its extensible module system, to Amazon Kinesis.
Here is a short video of how it looks like:
You will need to build your own release of logspout, including the logspout-kinesis module. To do so, create an empty Dockerfile based on
gliderlabs/logspout:master, and import this logspout-kinesis package into a new
modules.go file. The
gliderlabs/logspout base image will
ONBUILD COPY and replace the original
The following example creates a minimal logspout image capable of writing Dockers logs to Kinesis:
package main import ( _ "github.com/gliderlabs/logspout/httpstream" _ "github.com/gliderlabs/logspout/routesapi" _ "github.com/remind101/logspout-kinesis" )
Final step, build the image:
$ docker build -t mycompany/logspout .
Don’t forget to set up the environment variables and you’re all set! By attaching the docker Unix socket to the logspout container, you will then stream all the containers logs from your host.
When you type
emp log -a <app>, the Empire CLI will open a permanent HTTP connection to Empire which will then start streaming the logs back to the client. This means only the Empire instances need the Kinesis permissions to read from a stream. Empire gets the stream name for an app from its app ID, an internal Empire identifier specific to each app.
We are about to release a new version of Empire and its CLI emp. We will also update our official Empire AMI with logspout included. In the meantime, you can manually install logspout on your hosts and activate log streaming on Empire via setting up the
EMPIRE_LOGS_STREAMER variable to
kinesis. You can give it a try now on master and report any issues to https://github.com/remind101/logspout-kinesis/issues!
This article has to do with understanding user research from an engineering perspective. Whether you’re curious about the different types...