Centurion

A deployment tool for Docker. Takes containers from a Docker registry and runs them on a fleet of hosts with the correct environment variables, host volume mappings, and port mappings. Supports rolling deployments out of the box, and makes it easy to ship applications to Docker servers.

We're using it in our production infrastructure.

Centurion works in a two part deployment process where the build process ships a container to the registry, and Centurion ships containers from the registry to the Docker fleet. Registry support is handled by the Docker command line tools directly so you can use anything they currently support via the normal registry mechanism.

If you haven't been using a registry, you should read up on how to do that before trying to deploy anything with Centurion.

Commercial Docker Registry Providers:

• Docker, Inc. provides repositories, and hosts the main public Docker repository.
• Quay.io from the CoreOS team

Open-source:

• The Docker Distribution project, built and maintained by Docker. You host this yourself.

Status

This project still being maintained, but no new feature work is planned. The initial commit on GitHub contains one roll-up commit of all our internal code. But all internal development will now be on public GitHub. See the CONTRIBUTORS file for the contributors to the original internal project.

Installation

Centurion is a Ruby gem. It assumes that you have a working, modern-ish Ruby (1.9.3 or higher). On Ubuntu 12.04 you can install this with the ruby-1.9.1 system package, for example. On OSX this is best accomplished via rbenv and ruby-build which can be installed with Homebrew or from GitHub.

Once you have a running, modern Ruby, you simply:

$ gem install centurion

With rbenv you will now need to do an rbenv rehash and the commands should be available. With a non-rbenv install, assuming the gem dir is in your path, the commands should just work now.

Configuration

Centurion expects to find configuration tasks in the current working directory tree.

We recommend putting all your configuration for multiple applications into a single repo rather than spreading it around by project. This allows a central choke point on configuration changes between applications and tends to work well with the hand-off in many organizations between the build and deploy steps. If you only have one application, or don't need this you can decentralize the config into each repo.

It will look for configuration files in either ./config/centurion or ..

The pattern at New Relic is to have a configs repo with a Gemfile that sources the Centurion gem. If you want Centurion to set up the structure for you and to create a sample config, you can simply run centurionize once you have the Ruby Gem installed.

Centurion ships with a simple scaffolding tool that will setup a new config repo for you, as well as scaffold individual project configs. Here's how you run it:

$ centurionize -p <your_project>

centurionize relies on Bundler being installed already. Running the command will have the following effects:

• Ensure that a config/centurion directory exists
• Scaffold an example config for your project (you can specify the registry)
• Ensure that a Gemfile is present
• Ensure that Centurion is in the Gemfile (if absent it just appends it)

Any time you add a new project you can scaffold it in the same manner even in the same repo.

Writing configs

If you used centurionize you will have a base config scaffolded for you. But you'll still need to specify all of your configuration.

Configs are in the form of a Rake task that uses a built-in DSL to make them easy to write. Here's a sample config for a project called "radio-radio" that would go into config/centurion/radio-radio.rake:

namespace :environment do
  task :common do
    set :image, 'example.com/newrelic/radio-radio'
    host 'docker-server-1.example.com'
    host 'docker-server-2.example.com'
  end

  desc 'Staging environment'
  task :staging => :common do
    set_current_environment(:staging)
    env_vars YOUR_ENV: 'staging'
    env_vars MY_DB: 'radio-db.example.com'
    host_port 10234, container_port: 9292
    host_port 10235, container_port: 9293
    host_volume '/mnt/volume1', container_volume: '/mnt/volume2'
  end

  desc 'Production environment'
  task :production => :common do
    set_current_environment(:production)
    env_vars YOUR_ENV: 'production'
    env_vars MY_DB: 'radio-db-prod.example.com'
    host_port 22234, container_port: 9292
    host_port 23235, container_port: 9293
    command ['/bin/bash', '-c', '/path/to/server -e production']
  end
end

This sets up a staging and production environment and defines a common task that will be run in either case. Note the dependency call in the task definition for the production and staging tasks. Additionally, it defines some host ports to map, sets which servers to deploy to, and sets a custom command. Some configuration will be provided to the containers at startup time, in the form of environment variables.

Most of the DSL items (host_port, host_volume, env_vars, host) can be specified more than once and will append to the configuration. However, there can only be one command; the last one will take priority.

You can also assign lambdas to environment variables. The lambda function will be invoked by Centurion during deploy time with server_hostname as its only argument. The server_hostname yielded will be the same string as was specified in the host argument for the server currently being deployed to.

This is useful to assign a sticky identity for each of the containers in the deploy. For example, a hash mapping hostname to another string which is different on each host.

  desc 'Host-specific env vars'
  task :production => :common do
    env_vars MEMBER_ID: lambda do |hostname| 
        {
          'web-server-1.company.net' => 'machine1'
          'web-server-2.company.net' => 'machine2'
        }[hostname]
      end
  end

You can cause your container to be started with a specific DNS server IP address (the equivalent of docker run --dns 172.17.42.1 ...) like this:

  task :production => :common do
    set :dns, [ '172.17.42.1' ]
    # ...
  end

Container Names

This is the name that shows up in the docker ps output. It's the name of the container, not the hostname inside the container. By default the container will be named using the name of the project as the base of the name.

If you want to name your container something other than the project name, use the name setting. The actual name for the created container will have a random hex string appended, to avoid name conflicts when you repeatedly deploy a project:

  task :common do
    set :name, 'backend'
    # ...
  end

With this, the container will be named something like backend-4f692997.

Container Labels

You may add arbitrary labels to your containers by calling labels with a hash. The call is cumulative, so you may express patterns like:

namespace :environment do
  task :common do
    set :image, 'example.com/newrelic/radio-radio'
    host 'docker-server-1.example.com'
    labels team: 'radio-ops'
  end

  desc 'Staging environment'
  task :staging => :common do
    labels environment: 'radio-staging'
    env_vars YOUR_ENV: 'staging'
  end
end

This would result in the container having two labels, as shown in a docker inspect example:

  "Labels": {
    "team": "radio-ops",
    "environment": "radio-staging"
  }

Hash keys and values will be stringified, so you may pass any object with a #to_s method.

Container Hostnames

If you don't specify a hostname to use inside your container, the container will be given a hostname matching the container ID. This probably is good for a lot of situations, but it might not be good for yours. If you need to have a specific hostname, you can ask Centurion to do that:

set :container_hostname, 'yourhostname'

That will make all of your containers named 'yourhostname'. If you want to do something more dynamic, you can pass a Proc or a lambda like this:

set :container_hostname, ->(hostname) { "#{hostname}.example.com" }

The lambda will be passed the current server's hostname. So, this example will cause ".example.com" to be appended to the hostname of each Docker host during deployment.

If you want to restore the old behavior from Centurion 1.6.0 and earlier, you can do the following:

set :container_hostname, ->(hostname) { hostname }

That will cause the container hostname to match the server's hostname.

Network modes

You may specify the network mode you would like a container to use via:

set :network_mode, 'networkmode'

Docker (and therefore Centurion) supports one of bridge (the default), host, and container:<container-id> for this argument.

In host and container... network modes, you may specify a host_port, container_port mapping, however the port numbers will only be used for container health checks. The mapping itself, while still passed via the API, will be ignored by Docker.

PID modes

You may specify the PID mode you would like a container to use via:

set :pid_mode, 'pidmode'

Docker (and therefore Centurion) supports one of nothing (the default), host, and container:<container-id> for this argument.

CGroup Resource Constraints

Limits on memory and CPU can be specified with the memory and cpu_shares settings. Both of these expect a 64-bit integer describing the number of bytes, and the number of CPU shares, respectively.

For example, to limit the memory to 1G, and the cpu time slice to half the normal length, include the following:

memory 1.gigabyte
cpu_shares 512

For more information on Docker's CGroup limits see the Docker docs.

Adding Extended Capabilities

Additional kernel capabilities may be granted to containers, permitting them device access they do not normally have. You may specify these as follows:

add_capability 'SOME_CAPABILITY'
add_capability 'ANOTHER_CAPABILITY'
drop_capability 'SOMEOTHER_CAPABILITY'

You may also ask for all but a few capabilities as follows:

add_capability 'ALL'
drop_capability 'SOME_CAPABILITY'

For more information on which kernel capabilities may be specified, see the Docker docs.

Setting the security options

Some Docker platforms support container security overlays called seccomp. During container creation, you may specify security options to control the seccomp permissions.

To set a seccomp path:

add_security_opt 'seccomp=/path/to/seccomp/profile.json'

Or, to unblock all syscalls in a container:

add_security_opt 'seccomp=unconfined'

For more information on this argument, see the Docker docs.

Interpolation

Currently there a couple of special strings for interpolation that can be added to any env_var value in the DSL. %DOCKER_HOSTNAME% will be replaced with the current server's hostname in the environment variable at deployment time. Also %DOCKER_HOST_IP% will be replaced with the public IP address of the Docker server using a getaddrinfo call on the client.

Use TLS certificate

Centurion can use your existing Docker TLS certificates when using Docker with TLS support. In doing so you have 2 choices.

Your certificate files are in ~/.docker/

You just need to enable the tls mode as the following:

  task :production => :common do
    set :tlsverify, true
    # ...
  end

Centurion will only set the --tlsverify to true and Docker will read your certificate from the ~/.docker/ path.

Your certificate files are not in ~/.docker/

Given your files are in /usr/local/certs/ You have to set the following keys:

  task :production => :common do
    set :tlsverify, true
    set :tlscacert, '/usr/local/certs/ca.pem'
    set :tlscert, '/usr/local/certs/ssl.crt'
    set :tlskey, '/usr/local/certs/ssl.key'
    # ...
  end

Modify the paths as appropriate for your cert, ca, and key files.

Use SSH to connect beta

If your Docker server does not expose its HTTP service over TCP, you can instead talk to it via SSH.

This functions by creating a local Unix socket that forwards to the remote Docker Unix socket, so it requires that the user you connect as has access to the Docker socket without any sudo. Currently it also assumes that you authenticate via public key, so be sure that you have ssh-added your key to your SSH agent if it has a passcode.

You can configure it with a few options:

  task :common do
    set :ssh, true # enable ssh connections
    set :ssh_user, "myuser" # if you want to specify the user to connect as, otherwise your current user
    set :ssh_log_level, Logger::DEBUG # passed on to net/ssh, can be noisy; defaults to Logger::WARN
    set :ssh_socket_heartbeat, 5 # passed on to net/ssh (Net::SSH::Connection::Session#loop); defaults to 30
  end

Troubleshooting SSH connections

In some cases you may notice your SSH commands are completing successfully, but the connection isn't aware they are done. This will manifest as a tunneled command taking much longer than anticipated - minutes depending on your server's configuration. You may need to set :ssh_socket_heartbeat to a smaller number. This will check more frequently if the command has completed which can alleviate this issue, though it will consume more CPU as it wakes up its threads more frequently.

Deploying

Centurion supports a number of tasks out of the box that make working with distributed containers easy. Here are some examples:

Do a rolling deployment to a fleet of Docker servers

A rolling deployment will stop and start each container one at a time to make sure that the application stays available from the viewpoint of the load balancer. As the deploy runs, a health check will hit each container to ensure that the application booted correctly. By default, this will be a GET request to the root path of the application. The healthcheck endpoint is configurable by adding set(:status_endpoint, '/somewhere/else') in your config. The status endpoint must respond with a valid response in the 200 status range.

$ bundle exec centurion -p radio-radio -e staging -a rolling_deploy

Custom Health Check: You can use a custom health check by specifying a callable object (anything that responds to :call), e.g. a Proc, lambda, or method. This method will be invoked with the host url, the port that needs to be checked, and the specified endpoint(via set(:status_endpoint, '/somewhere/else')). If the port is ready, health check should return a truthy value, falsey otherwise. Here's an example of a custom health check that verifies that an elasticsearch node is up and has joined the cluster.

def cluster_green?(target_server, port, endpoint)
  response = begin
    Excon.get("http://#{target_server.hostname}:#{port}#{endpoint}")
  rescue Excon::Errors::SocketError
    warn "Elasticsearch node not yet up"
    nil
  end

  return false unless response
  !JSON.parse(response)['timed_out']
end

task :production => :common do
  set_current_environment(:production)
  set :status_endpoint, "/_cluster/health?wait_for_status=green&wait_for_nodes=2"
  health_check method(:cluster_green?)
  host_port 9200, container_port: 9200
  host 'es-01.example.com'
  host 'es-02.example.com'
end

Rolling Deployment Settings: You can change the following settings in your config to tune how the rolling deployment behaves. Each of these is controlled with set(:var_name, 'value'). These can be different for each environment or put into a common block if they are the same everywhere. Settings are per-project.

• rolling_deploy_check_interval => Controls how long Centurion will wait after seeing a container as up before moving on to the next one. This should be slightly longer than your load balancer check interval. Value in seconds. Defaults to 5 seconds.
• rolling_deploy_wait_time => The amount of time to wait between unsuccessful health checks before retrying. Value in seconds. Defaults to 5 seconds.
• rolling_deploy_retries => The number of times to retry a health check on the container once it is running. This count multiplied by the rolling_deployment_wait_time is the total time Centurion will wait for an individual container to come up before giving up as a failure. Defaults to 24 attempts.
• rolling_deploy_skip_ports => Either a single port, or an array of ports that should be skipped for status checks. By default status checking assumes an HTTP server is on the other end and if you are deploying a container where some ports are not HTTP services, this allows you to only health check the ports that are. The default is an empty array. If you have non-HTTP services that you want to check, see Custom Health Checks in the previous section.

Deploy a project to a fleet of Docker servers

This will hard stop, then start containers on all the specified hosts. This is not recommended for apps where one endpoint needs to be available at all times. It is fast.

$ bundle exec centurion -p radio-radio -e staging -a deploy

Deploy a bash console on a host

This will give you a command line shell with all of your existing environment passed to the container. The CMD from the Dockerfile will be replaced with /bin/bash. It will use the first host from the host list.

$ bundle exec centurion -p radio-radio -e staging -a deploy_console

Repair unhealthy docker containers

This will preform a health check on each host using rolling deployment health check settings and redeploy to the host if a health check fails.

$ bundle exec centurion -p radio-radio -e staging -a repair

List all the tags running on your servers for a particular project

Returns a nicely-formatted list of all the current tags and which machines they are running on. Gives a unique list of tags across all hosts as well. This is useful for validating the state of the deployment in the case where something goes wrong mid-deploy.

$ bundle exec centurion -p radio-radio -e staging -a list:running_container_tags

List all the containers currently running for this project

Returns a (as yet not very nicely formatted) list of all the containers for this project on each of the servers from the config.

$ bundle exec centurion -p radio-radio -e staging -a list:running_containers

List registry images

Returns a list of all the images for this project in the registry.

$ bundle exec centurion -p radio-radio -e staging -a list

Registry

Centurion needs to have access to some registry in order to pull images to remote Docker servers. This needs to be either a hosted registry (public or private), or Dogestry.

Access to the registry

If you are not using either Dogestry, or the public registry, you may need to provide authentication credentials. Centurion needs to access the Docker registry hosting your images directly to retrive image ids and tags. This is supported in both the config file and also as command line arguments.

The command line arguments are:

• --registry-user => The username to pass to the registry
• --registry-password => The password

These correspond to the following settings:

• registry_user
• registry_password

Alternative Docker Registry

Centurion normally uses the built-in registry support in the Docker daemon to handle pushing and pulling images. But Centurion also has the ability to use external tooling to support hosting your registry on Amazon S3. That tooling is from a project called Dogestry. We have recently improved that tooling substantially in coordination with the Centurion support.

Dogestry uses the Docker daemon's import/export functionality in combination with Amazon S3 to provide reliable hosting of images. Setting Centurion up to use Dogestry is pretty trivial:

1. Create an S3 bucket and download the credentials to let you access the bucket. Generally these are IAM user keys.
2. Install Dogestry binaries on the client from which Dogestry is run. Binaries are provided in the GitHub release.
3. Add the settings necessary to get Centurion to pull from Dogestry. A config example is provided below:

namespace :environment do
  task :common do
    registry :dogestry                       # Required
    set :aws_access_key_id, 'abc123'         # Required
    set :aws_secret_key, 'xyz'               # Required
    set :s3_bucket, 'docker-images-bucket'   # Required
    set :s3_region, 'us-east-1'              # Optional
  end
end

TLS with Dogestry: Because this involves a little passing around of both settings and environment variables, there are a couple of things to verify to make sure everything is passed properly between Centurion and Dogestry. If your keys have the default names and are in located in the path represented by DOCKER_CERT_PATH in your environment, this should just work. Otherwise you'll need to be sure to set :tlsverify, true and also set the TLS cert names as decribed above.

Development

Centurion supports a few features to make development easier when building your deployment tooling or debugging your containers.

Overriding Environment Variables

Sometimes when you're doing development you want to try out some configuration settings in environment variables that aren't in the config yet. Or perhaps you want to override existing settings to test with. You can provide the --override-env command line flag with some overrides or new variables to set. Here's how to use it:

$ centurion -e development -a deploy -p radio-radio --override-env=SERVICE_PORT=8080,NAME=radio

Centurion is aimed at repeatable deployments so we don't recommend that you use this functionality for production deployments. It will work, but it means that the config is not the whole source of truth for your container configuration. Caveat emptor.

Exporting Environment Variables Locally

Sometimes you need to test how your code works inside the container and you need to have all of your configuration exported. Centurion includes an action that will let you do that. It exports all of your environment settings for the environment you specify. It then partially sanitizes them to preserve things like rbenv settings. Then it executes /bin/bash locally.

The action is named dev:export_only and you call it like this:

$ bundle exec centurion -e development -p testing_project -a dev:export_only
$ bundle exec rake spec

It's important to note that the second line is actually being invoked with new environment exported.

Future Additions

We're currently looking at the following feature additions:

• etcd integration for configs and discovery
• Add the ability to show all the available tasks on the command line

Contributions

Contributions are more than welcome. Bug reports with specific reproduction steps are great. If you have a code contribution you'd like to make, open a pull request with suggested code.

Note that PR's and issues are reviewed every ~2 weeks. If your PR or issue is critical in nature, please reflect that in the description so that it receives faster attention.

Pull requests should:

• Clearly state their intent in the title
• Have a description that explains the need for the changes
• Include tests!
• Not break the public API
• Add yourself to the CONTRIBUTORS file. I might forget.

If you are simply looking to contribute to the project, taking on one of the items in the "Future Additions" section above would be a great place to start. Ping us to let us know you're working on it by opening a GitHub Issue on the project.

By contributing to this project you agree that you are granting New Relic a non-exclusive, non-revokable, no-cost license to use the code, algorithms, patents, and ideas in that code in our products if we so choose. You also agree the code is provided as-is and you provide no warranties as to its fitness or correctness for any purpose

Copyright (c) 2014-2017 New Relic, Inc. All rights reserved.