A High Availability Phoenix and A/B Deployment Framework using Docker

Currently co-authoring a book on Docker: Get 39% off with the code 39miell

tl;dr

A four-step worked example of a process for creating a phoenix-deployed service fronted by haproxy, with two swappable backends deployed using Docker containers, enabling continuous delivery with minimal effort.

Introduction

I’ve long been interested in Phoenix deployment, and Docker as a means of achieving it. I’ve built my own Phoenix deployment framework which I use to automate deployment and auto-rebuild and deploy from scratch daily.

Phoenix Deployment?

Phoenix deployment is the principle of rebuilding ‘from scratch’ rather than updating an environment. Tools like Chef and Puppet are great for managing long-lived servers, but nothing beats regular rebuilds and deployments for ensuring you can reason about your environments.

I’ve been using Phoenix deployment to rebuild applications and services on a daily basis regardless of whether changes have been made. See here and here for previous posts on the subject.

Architecture

If you’re refreshing a service though you generally need to minimise downtime while. To achieve this, I use HAProxy to provide a stable endpoint for two backends – ‘old’ and ‘new’, as this figure shows:

Image A and Image B are constructed from scratch each time using ShutIt, an automation tool designed with Phoenix deployments in mind.

 

Worked Example

Here’s a worked example. In it you will create an image that acts as a simple echo server, and then change the code and redeploy the service as a server that converts strings to hex.

Note: tested on a Digital Ocean 14.0.4 image.

 

1) Install pre-requisites:

sudo apt-get update && apt-get install python-pip git docker
sudo pip install shutit

2) Create phoenix build

Create the skeleton script, accepting the defaults:

user@host$ shutit skeleton

# Input a new directory name for this module.
# Default: /tmp/shutit_cohort

# Input module name.
# Default: shutit_cohort

# Input a unique domain.
# Default: imiell.shutit_cohort

# Input a delivery method from: ('docker', 'dockerfile', 'ssh', 'bash').
# Default: docker

docker = build within a docker image
dockerfile = call "shutit build" from within a dockerfile
ssh = ssh to target and build
bash = run commands directly within bash

================================================================================
Run:
cd /tmp/shutit_cohort/bin && ./build.sh
to build.
An image called shutit_cohort will be created
and can be run with the run.sh command in bin/.
================================================================================

 

3) Edit build

In this step you’re going to set up your echo server application as a simply python script embedded in a container image.

Go to the directory just created, eg for the above output:

cd /tmp/shutit_cohort

and open the python file in there:

vi shutit_cohort.py

and change the build method so it looks like this:

def build(self,shutit):
[...]
    # shutit.set_password(password, user='')
    # - Set password for a given user on target       
    shutit.install('socat')
    shutit.send(r'''echo socat tcp-l:80,fork exec:/bin/cat > /echo.sh''')
    shutit.send('chmod +x /echo.sh')

The line:

shutit.install('socat')

ensures that socat is installed on the container, and the next lines:

    shutit.send(r'''echo socat tcp-l:80,fork exec:/bin/cat > /echo.sh''')
    shutit.send('chmod +x /echo.sh')

creates the file ‘/echo.sh’ on the container as an executable using socat that acts as an echo server.

You want your service to run the python script, so change the file ‘bin/run.sh’ and change the last line from this:

${DOCKER} run -d --name ${CONTAINER_NAME} ${DOCKER_ARGS} ${IMAGE_NAME} /bin/sh -c 'sleep infinity'

to this:

${DOCKER} run -d --name ${CONTAINER_NAME} ${DOCKER_ARGS} ${IMAGE_NAME} /bin/sh -c '/echo.sh'

ie replace ‘sleep forever’ with your ‘/echo.sh’ command.

Note: Will use ports 8080-8082.

If these are used for other services, change the ports in phoenix.sh

4) Build and deploy the service

OK, we’re ready to go.

cd bin
sudo ./phoenix.sh

Kicks off the build and deploys the service. It builds and run the HAProxy server and the image that acts as back end ‘A’.

# CONTAINER_ID: 37352b9918bd08b843f2c5174266e1af199b6d05520551b4f9f0489342995618
# BUILD REPORT FOR BUILD END phoenix_imiell_1440614873.89.890216
###############################################################################

Build log file: /tmp/shutit_root/phoenix_imiell_1440614873.89.890216/shutit_build.log
/tmp/shutit_coolly/bin
f500e552cdd20445266ed4d6fa2d1ba3d55ca9845ea39744fc1c8ba1dd96a762

docker ps -a shows our two servers: haproxy taking requests on the host network, and passing it to the backend on 8081:

$ docker ps -a | grep shutit
f500e552cdd2  shutit_coolly          "/bin/sh -c /echo.sh"     0.0.0.0:8081->80/tcp   shutit_coolly
b28abcb76612  shutit_coolly_haproxy  "haproxy -f /usr/loca"                           shutit_coolly_haproxy

Now test your echo server:

imiell@phoenix:/tmp/shutit_coolly/bin$ telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello phoenix
hello phoenix

Note: You only need sudo if you need sudo to run docker on your host.

5)Iterate and re-deploy

Now you’re going to change the server and redeploy. We want to use cat’s -A flag to output more details when echoing, so change the socat line in the python script to:

shutit.send(r'''echo socat tcp-l:80,fork exec:'/bin/cat -A' > /echo.sh''')

and re-run the phoenix.sh as  you did before. When done, docker ps -a now shows the container running on port 8082 (ie the ‘B’ port):

$ docker ps -a | grep shutit
af0abdd3abc9 shutit_coolly         "/bin/sh -c /echo.sh"  0.0.0.0:8082->80/tcp shutit_coolly
b28abcb76612 shutit_coolly_haproxy "haproxy -f /usr/loca"                      shutit_coolly_haproxy

and to verify it’s worked:

$ telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello phoenix
hello phoenix^M$

Conclusion

By putting this code into git you can easily create a continuous deployment environment for your service that doesn’t interfere with other services, and is easy to maintain and keep track of.

I use this framework for various microservices I use on my home server, from databases I often want to run queries on to websites I manage. And this blog :)