Our log streaming solution based on Amazon Kinesis

Today, we are open sourcing our log streaming solution for Empire.

Since day one, we’ve been able to stream our logs to analytics and search platforms such as Sumologic, but we were missing a simple, immediate solution for real-time inspection of logs, something as simple as emp log -a <app>, a la Heroku.

Heroku’s solution, Logplex is open source, but is difficult to setup and operationally complex - two things we try to avoid in Empire.

Introducing Logspout-Kinesis

Logspout-Kinesis is a module for logspout, that buffers and streams logs to Amazon Kinesis. It is capable of:

• streaming to one or multiple kinesis streams
• ignoring logs from specific containers
• creating and tagging a stream for an app

From the stream names, to the tags and the partition keys, everything is configurable via environment variables and go templates. You can use it however you want.

Since we are building 12-factor apps at Remind, we have always been logging to stdout, and logspout has been working extremely well for us. It attaches to all containers on a host, then routes their logs wherever we want. We use it to send our logs to Sumologic, and now thanks to its extensible module system, to Amazon Kinesis.

Here is a short video of how it looks like:

How to set it up

You will need to build your own release of logspout, including the logspout-kinesis module. To do so, create an empty Dockerfile based on gliderlabs/logspout:master, and import this logspout-kinesis package into a new modules.go file. The gliderlabs/logspout base image will ONBUILD COPY and replace the original modules.go.

The following example creates a minimal logspout image capable of writing Dockers logs to Kinesis:

In modules.go:

package main

import (
  _ "github.com/gliderlabs/logspout/httpstream"
  _ "github.com/gliderlabs/logspout/routesapi"
  _ "github.com/remind101/logspout-kinesis"
)

In Dockerfile:

FROM gliderlabs/logspout:master

Final step, build the image:

$ docker build -t mycompany/logspout .

Don’t forget to set up the environment variables and you’re all set! By attaching the docker Unix socket to the logspout container, you will then stream all the containers logs from your host.

What about consuming the logs?

When you type emp log -a <app>, the Empire CLI will open a permanent HTTP connection to Empire which will then start streaming the logs back to the client. This means only the Empire instances need the Kinesis permissions to read from a stream. Empire gets the stream name for an app from its app ID, an internal Empire identifier specific to each app.

What now?

We are about to release a new version of Empire and its CLI emp. We will also update our official Empire AMI with logspout included. In the meantime, you can manually install logspout on your hosts and activate log streaming on Empire via setting up the EMPIRE_LOGS_STREAMER variable to kinesis. You can give it a try now on master and report any issues to https://github.com/remind101/logspout-kinesis/issues!