Docker – One Year On

Introduction

It’s just over a year since I first heard of Docker, and I’ve been using it avidly one way or another ever since. A year seems like a good time to look back over what it’s done for us at $corp compared to what I’d hoped.

Docker is such a flexible technology it’s fascinating to see how different people view it and run with it. Some see it as “just a packaging tool” (as if that were not a significant thing in itself), others as a means to delivering on the microservices dream, still others as a way of saving CPU cycles at scale.

Our use case was very much about improving predictability in a sprawling decades-old codebase for the development, QA, support and testing cycles. As far as we’re concerned, it’s definitely delivered, but we’ve had to go off-piste a bit.

Main Achievements

The tl;dr is that using Docker has saved us significant amounts of time, and in addition enabled us to do things considered previously impractical. There are too many to list (available on request!) but here are some highlights:

• Our biggest dev team (~50 devs) has switched from a “shared dev server” development paradigm, to dev’ing on local machines from Docker images built daily. Others are now following.
• QA processes have improved significantly – for example, automated endpoint testing was a trivial addition to the daily-built dev env, complete with emails sent to the testers with logs etc..
• We have reduced the reliance and load on a centralised IT team always torn between stability and responsiveness (“why can’t we just upgrade python on the shared servers!?”)
• CI is now using a centralised reference image as a Jenkins slave, again reducing dependence on IT for changes. Docker’s image layering provides a neat saving in on-premise disk space also.
• Multi-node regression testing environments can be easily deployed on an engineer’s laptop (we use Skydock for this).
• Escrow build auditing processes which were previously onerous now come “for free”, as we phoenix-deploy dev environments.

Central to all these has been using ShutIt to enable us to encapsulate our complex build and dev needs into a single point of reference. Uptake of ShutIt  has been far better than with traditional CM tools like Puppet or Ansible, mainly due to speed-to-market rather than maturity of the tool. Relatively little training is needed to slot your changes into the ShutIt builds and regression test it; everyone can pitch in quickly.

Lessons Learned

Getting traction for a new technology with software development house like ours (500+ engineers in various locations etc) is non-trivial. Here are some lessons learned:

• Find a few motivated people and give them space to make it work
• Focus them on a single well-defined problem (in our case: canned dev environments)
• This problem should have a clear benefit to the business

In terms of the Docker systems, these were some of the things we learned as we went:

• Dockerfiles did not scale with our image need. This may be because we’re “doing it wrong” (ie our software architecture is too monolithic), but Dockerfiles were plain impractical for us, so we effectively dropped them in favour of ShutIt
• Having pre-built vanilla base images were very useful in getting traction. Being able to hand people a useful and up-to-date image to start working on was seen as a win. A monthly cadence for the base image worked for us, while dailies were essential for the development team.
• Building on this known base has facilitated several projects that otherwise may have taken longer to get going.
• The Docker community is very strong and very useful – without Jerome Petazzioni‘s blogs and help on Docker mailing lists we’d have been stymied a few times.
• Running your containers within a VM can help reduce nerves elsewhere in your business about security beyond isolation, or managing shared compute resource (until that space matures in the Docker ecosystem).
• Managing your own registry so that people can play with it is a good idea if you want privacy, but make sure you have a plan for when the disk space usage blows up!

Future

As Docker usage grows throughout or organisation there are several things we have our eyes on over the horizon.

• CoreOS looks like a very promising technology for managing Docker resources, and we’re playing with this very informally
• Enterprise support is something we’ve explored, but since we’ve not gone near live with this yet we’ve not pursued it. I’d be interested to know what people’s experience with it is though

Over to You

Are your experiences with getting Docker out there similar, or completely different? I’d be delighted to hear from you (@ianmiell / ian.miell@gmail.com)

Recently someone brought to my attention this issue on Github.

https://github.com/taigaio/taiga-scripts/issues/3

It’s a common problem. Awesome-looking project, how exactly do I install it to play with it?

This is a perfect use case for ShutIt.

I built it by setting up a skeleton directory, which creates a standalone ShutIt module:

./shutit skeleton /path/to/shutit/library/taigaio taigaio shutit.tk

which gave me the boilerplate to produce the build section here:

https://github.com/ianmiell/shutit/blob/master/library/taigaio/taigaio.py#L10

You can see the ShutIt API is fairly intuitive – you call commands like login, send, multisend, run_script and logout on the shutit object to perform actions within the bash session that’s set up for you.

Once written, you can test with:

$ cd /path/to/shutit/library/taigaio/bin
$ ./build.sh

Then, to run it:

$ docker run -i -t -p 127.0.0.1:8000:8000 -p 127.0.0.1:8001:8001 taigaio bash -c '/root/start_postgres.sh && /root/start_taiga.sh && echo READY! && sleep 3000d'

Navigate to http://localhost:8000 and login as admin/123123

If you just want to get going:

$ docker run -i -t -p 127.0.0.1:8000:8000 -p 127.0.0.1:8001:8001 imiell/taigaio bash -c '/root/start_postgres.sh && /root/start_taiga.sh && echo READY! && sleep 3000d'

Wait to see “READY!” and navigate to http://localhost:8000 and login as admin/123123

Voila:

Using ShutIt and Docker to play with AWS (Part Two)

Previously I showed a very basic way of using ShutIt to connect to AWS.

I’ve taken this one step further so now there is a template for automating:

• killing any AWS instance you have running
• provisioning a new instance
• logging onto the new instance
• installing docker on it
• pulling and running your image

All you will need is a .pem file and to know the security group you want to use.

I’m assuming you already have an AWS account on the free tier and have nothing running on it, or one throwaway instance we’re going to kill.

Here are the steps to get going, from an ubuntu:14.04 basic install:

1) Basic installs

apt-get update
apt-get install -y docker.io python-pip
git clone https://github.com/ianmiell/shutit.git
cd shutit
pip install -r requirements.txt 
mkdir -p ~/.shutit && touch ~/.shutit/config && chmod 600 ~/.shutit/config 
vi ~/.shutit/config

2)Edit config for AWS

Then edit the file as here, changing the bits in CAPS as necessary:

[shutit.tk.aws.aws]
access_key_id:YOUR_AWS_ID
secret_access_key:YOUR_AWS_KEY
# region, eg 
# region:eu-west-1
region:YOUR_AWS_REGION
[shutit.tk.aws_example.aws_example]
# Your pem filename, eg if your pem file is called: yourpemname.pem
# pem_name:yourpemname 
pem_name:YOUR_PEM_NAME

3) Place your .pem file in the context’s pem directory:

cp /path/to/yourpemname.pem examples/aws_example/context/yourpemname.pem

4) Run it

cd examples/aws_example_provision
../../shutit build --shutit_module_path ../../library:.. --interactive 0

And wait.

Once you’ve seen that that works, you can now try changing it to automate the setup of an app on AWS.

You can start by uncommenting the lines here in aws_example_provision.py:

 shutit.send('sudo yum install -y docker')
 shutit.send('sudo service docker start')
 # Example of what you might want to do. Note that this won't work out
 # of the box as the security policy of the VM needs to allow access to
 # the relevant port.
 #shutit.send('sudo docker pull training/webapp')
 #shutit.send('sudo docker run -d --net=host training/webapp')
 # Exit back to the "real container"
 shutit.logout()

And running the build again.

An edited talk I gave a while back on Docker and ShutIt:

 

I’m using ShutIt to play with AWS at the moment.

I can leverage the core libraries to easily build on top with my secret data and store in my source control system, and I’ll show how you can do this here.

Firstly, there’s a core aws library that takes care of installing the aws command line tool which is part of the ShutIt libraries:

Shutit AWS module

ShutIt AWS build script

It contains the ability to configure the AWS access token, but obviously we don’t want to store that in the core library.

The solution to this to create my own module which effectively inherits from that generic solution, adding my pems and configuring for access.

/space/git/shutit/shutit skeleton /my/git/path/ianmiellaws ianmiellaws my.domain
cd /my/git/path/ianmiellaws
mv /path/to/pems context/
cd /my/git/path/ianmiellaws/
vi configs/
vi ianmiellaws.py
./test.sh

The bits in bold were the bits edited in the above vi edit:

imiell@lp01728:/space/git/work/notes/aws/ianmiellaws$ cat ianmiellaws.py
from shutit_module import ShutItModule

class ianmiellaws(ShutItModule):

    def is_installed(self,shutit):        
        return False

    def build(self,shutit):
        shutit.send_host_file('t2.pem','context/pems/t2.pem')     
        return True


def module():
    return ianmiellaws(
        'my.domain.ianmiellaws.ianmiellaws', 1159697827.00,
        description='',
        maintainer=''
        depends=['shutit.tk.setup','shutit.tk.aws.aws']       
)

In the config file edit I put (replacing the stuff in caps with my details):

[my.domain.aws.aws]
access_key_id:MYKEYHERE
secret_access_key:MYSECRETACCESSKEYHERE
region:MYREGION
output:

Then build it:

/path/to/shutit build -m /path/to/shutit/library

Then run it:

docker run -t -i ianmiellaws /bin/bash

and you should be able to access your AWS services from wherever you have the container.

In the next posts I’ll show how to build on top of this to write a module to automatically provision an AWS instance and run a docker service on it.

I’ve posted previously on phoenix deployment, and thought I’d share an incident that happened that was resolved more easily because it was caught early.

Got a ping that the site was down, so looked into it.

Tried rebuilding the site using ShutIt and discovered quickly that apt-gets were failing.

So what changed?

I checked the network (fine), and then tried to repro on a simple ubuntu:12.10 image. Same there. I did the same on a completely different network. Tried hitting the apt urls directly – white screen.

A quick “ask around” later and I figure out that support for these ubuntu repos had been withdrawn.

So then I could work on an upgrade relatively quickly, and I had yesterday’s container to compare differences. In the meantime apache’s default security settings had changed, so that was quite painful.

But not as painful as it could have been, thanks to PD.

I wrote a website in my spare time (themortgagemeter.com) a few years ago for a family member’s business tracking mortgage rates over time in real time.

As an experiment I wrote it in such a way that the entire database would be backed up to BitBucket daily (as well as the running code on the live box). This allowed me to easily pull the repo, and dev straight off master wherever I was developing. It was well worth the investment; much time was saved in easy dev environment creation.

When Docker came along, I thought this would be an ideal opportunity to encapsulate the entire site in Docker to further this efficiency. Since there was configuration and interactions to manage, I used ShutIt to store the configuration and build steps.

Once it was containerized (or Shut), it was a short step to implement phoenix deployment.

Phoenix deployment is the principle of rebuilding rather than upgrading. Fortunately ShutIt made it easy to implement this.

Here is the script:

$ cat phoenix.sh 
/space/git/shutit/shutit build --shutit_module_path /space/git/shutit/library --image_tag stackbrew/ubuntu:raring
docker.io tag $(sudo docker commit $(sudo docker ps -a | grep -v themortgagemeter | grep raring | awk '{print $1}')) themortgagemeter
docker.io rm -f themortgagemeter
docker.io run -t -d -h themortgagemeter --name themortgagemeter -p 80:80 themortgagemeter

So now I have the peace of mind of knowing that whatever is running there now was built from scratch today.

With some simple tests (part of the ShutIt build lifecycle) and external monitoring (using Uptime Robot) I can be sure it will not suffer from bit-rot.

Docker, ShutIt and the Perfect 2048 Game

(4 – Halfway There)

Anyone care to guess the score at this point?

 

Docker, ShutIt and the Perfect 2048 Game (3 – Brute Force Escapes)

 

Now that I’m getting near the end of the highest tile on 2048, the air is getting thin.

I often get into a state like this:

where I need a four – not a two – to drop in on the top.

Since fours are less frequent than twos, I have to start up, try, quit, and repeat multiple times, which gets quite tedious.

So I automated it using visgrep.

visgrep is a seemingly obscure tool (part of the xautomation package on ubuntu) to search for images within other images. I won’t go into its use here (it’s _not_ user-friendly and in-depth documentation is perhaps kept by monks somewhere), but there is a man page.

I’ve used it in the commented-out sections here:

https://github.com/ianmiell/shutit/blob/master/library/win2048/resources/start_win2048.sh

To use it, uncomment the lines, and put the appropriate key code in, save the container and run it as below:

host_prompt$ while [ 1 ]; do sudo docker run -t -i -p 5901:5901 -p 6080:6080 -e HOME=/root imiell/2048_left /root/start_vnc.sh; sleep 5; done

New 'b80055fbbeff:1 ()' desktop is b80055fbbeff:1

Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/b80055fbbeff:1.log

Did you expose ports 5901 and 6080?
If so, then vnclient localhost:1 should work.
18874476
307,129 0
307,130 0
306,131 0
307,131 0
308,131 0
307,132 0
307,133 0
0
FAIL
[about output loops n times]

New '3fd43021ea4a:1 ()' desktop is 3fd43021ea4a:1

Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/3fd43021ea4a:1.log

Did you expose ports 5901 and 6080?
If so, then vnclient localhost:1 should work.
16777324
0
OK
container_prompt$

So when I see an “OK” and a prompt within the container I know I can vnc in and continue playing.

The time saved by doing this is significant.

I’m not really a game player, but I imagine this principle could be applied to a lot of games. Also, the process could be made much more slick, as this is still very much by-hand.