CI as Code Part II: Stateless Jenkins With Dynamic Docker Slaves

Introduction

This article continues on from part one of this series, which looks at ‘CI as code’ using Docker to set up isolated and reproducible phoenix deployments of Jenkins deployments

Here I add dynamic Docker containers as on-demand Jenkins nodes in a Docker cloud.

Code

This is now baked into v2.0 of the git repo.

Video

Here’s a video of the stateless setup of the Docker cloud, and the job ‘docker-test’ which dynamically provisions a Docker container to run as a Jenkins slave.

What it does

  1. Starts up Jenkins container with a server config config.xml preloaded
  2. ‘jenkinssetup’ container waits for Jenkins to be up and ready
  3. Sets up global credentials
  4. Updates Jenkins’ config.xml with the credentials id
  5. Restart Jenkins and wait for jenkins to be ready
  6. Kick off install of plugins
  7. Periodically restart Jenkins until plugins confirmed installed
  8. Upload job configurations

 

Details

The Docker plugins for Jenkins are generally poorly documented and fiddly to set up. And between them there’s quite a few, so the Docker options in a job available can get quite confusing. This took a little bit of trial and error before I could reliably get it to work.

To allow dynamic Docker provisioning, I used the standard docker plugin, mainly because it was the only one I ended up getting working with my Jenkins-in-docker-compose approach.

To get a dynamic on-demand Docker instance provisioned for every build, you have to set up a Docker cloud with the details of the Docker host to contact to spin up the container. This cloud is given a label, which you use in your job to specify that it should be run in a Docker container.

Currently co-authoring a book on Docker:

Get 39% off with the code 39miell2

dip

 

Note: If you want to recreate this you must have an opened-up Docker daemon.
See here for a great guide on this. Once that’s done you may need to change the
docker host address in the docker.xml field to point to your opened up Docker
daemon. Usually this is with the IP address outputted from ‘ip route’ in your
running containers. The default in the git repo is fine, assuming you have opened
it up on port 4243.

 

Advertisements

CI as Code Part I: Stateless Jenkins Deployments Using Docker

Currently co-authoring a book on Docker:

Get 39% off with the code 39miell2

dip

I don’t know about you, but I’ve always been uncomfortable with Jenkins’ apparent statefulness. You set up your Jenkins server, configure it exactly as you want it, then DON’T TOUCH IT.

For an industry apparently obsessed with ‘infrastructure/environments/whatever as code’ this is an unhappy state of affairs.

I’d set up a few Jenkins servers, thrown some away, re-set them up, and it always seemed a wasteful process, fraught with forgetfulness.

Fortunately I now have a solution. With a combination of Docker, Python’s Jenkins API modules, the Jenkins job builder Python module, and some orchestration using docker-compose, I can reproduce my Jenkins state at will from code and run it in isolated environments, improving in iterable, track-able steps.

Here’s a video of it running:

This example sets up:

  • a vanilla Jenkins instance via a Docker container
  • a simple slave node
  • a simple docker slave node
  • a container that sets up Jenkins with:
    • jobs
      • a simple echo command with no triggers
      • a docker echo command triggered from a github push
    • credentials
    • plugins

The code is here. I welcome contributions, improvements, suggestions and corrections.

To run it yourself, ensure you have docker-compose installed:

git clone https://github.com/ianmiell/jenkins-phoenix
cd jenkins-phoenix
git checkout tags/v1.0
./phoenix.sh

 

jenkins_stateless (4)

HN Discussion here.

See here for Part II, where dynamic Docker slaves are added.

Docker Ecosystem Rosetta Stones

Navigating the Docker ecosystem can be confusing at the best of times.

I’ve been involved in numerous conversations where people have confused Indexes, Registries, Commits, diffs, pushes, Pods, Containers, Images… the list goes on.

I’ve put together these references to try and help with these confusions. Please get in touch if anything seems amiss, or you have any suggestions for improvements.

Reference is kept here.

Docker

Container A running instance of a Docker image
Image A tagged history of layers make up an image
Layer A set of file-level diffs. See diff
Privileged Containers are privileged if they run with full root powers. Normally, root within the container has a reduced set of capabilities
Registry A registry is a hosted service containing repositories of images, which responds to the Docker registry API
Swarm Docker Swarm is a clustered set of Docker nodes connected and managed by a Swarm manager running on each node

Docker vs Git

Docker Git
Blob Most often seen in the context of registries (API version 2) where the objects managed by the registry are stored in binary objects The git index points to blobs that refer to content making up the repository’s history
Commit Takes the differences in the container you reference to the last image, and creates a new layer. The added layer and all previous layers constitutes a new image Takes the changes in your index and stores them in your local repository with a message
Diff Gives you the files added, updated, or deleted in a layer Gives a line-by-line diff of files between two revisions in the git repository’s history
History The layers that make up an image (in order) constitute an image’s history The ordered previous revisions in a repository
Hub DockerHub is an index of images and Dockerfiles managed by Docker.inc. GitHub is a popular central point of reference for software projects that use git
Index Apparently deprecated term for a registry A binary file in a git repository that stores references to all the blobs that make up a repository
Pull Images are pulled from registries to your local host Get a set of commits and apply them to your local repository
Push Committed and tagged images can be pushed to registries Commits can be pushed to remote repositories
Repository A collection of images distinguished by tag, eg docker.io/ubuntu A git project, ie the folder you might pull from a git remote
Remote N/A, though if someone uses this term, they probably mean registry A repository stored in another location, eg on GitHub or bitbucket.org
Revision N/A See commit
Tag Tags are applied to repositories to distinguish different images, eg docker.io/ubuntu:14.04 is a different image from docker.io/ubuntu:15.04, but both are in the same repository A reference to a git repository in a particular state

Kubernetes

Endpoint IP and port that accepts TCP or UDP flows
Kube-proxy Receives information about changes to services and endpoints
Kubelet See Replication Controller
Replication Controller Container supervisor. One of these runs on each host, ensuring the correct pods are running and in the appropriate number
Controller Manager Orchestrates replication controllers
Pod A grouping of containers that run on one host, and share volumes and a network stack (ie including IP and ports). Pods can run as one-offs, while long-running services should be spun up by replication controllers
Service An abstracted endpoint for a set of pods or other endpoint

OpenShift

Build Controller Component that manages the building of Docker images ready for deployment
Deployment Controller Component that manages the deployment of Docker images to pods on nodes in the cluster
Image Stream A set references to other images (and/or image streams). This provides a virtual view of related images, which allow operational control of events when any of the referenced image (and/or image streams) are changed. These events might trigger a rebuild or redeployment of a build.
Route A DNS-resolveable endpoint that maps to an ip address and port
Image Stream Tag Image Streams can be tagged in a similar way to Docker image tags

 

Docker vs Kubernetes vs OpenShift

Docker Kubernetes OpenShift
Cluster A cluster is a set of Nodes that run Docker Swarm  A Kubernetes Cluster is a set of master nodes and minions An OpenShift Cluster is a set of OpenShift ‘master nodes’ plus a set of OpenShift nodes
Label Name value pair applied to an object, eg an image or container Name value pair applied to an object, eg pod or node Name value pair applied to an object, eg route
Master The Swarm Node acting as the elected master The node or nodes that act as the cluster master, keeping track of centralised information using etcd nodes The node or nodes that act as the cluster master, keeping track of centralised information using etcd nodes
Minion N/A Node on which Pods can be run Rarely used, but would correspond to an OpenShift Node
Namespace Kernel facility to allocate an isolated instance of a global resource, eg filesystem or network. Docker is partly a product that orchestrates these isolated components in a consumable way Isolated sets of resources for management purposes Isolated sets of resources for management purposes
Node A host within a Swarm cluster A host within a Kubernetes cluster A host within an OpenShift cluster
Project N/A N/A Extension of Kubernetes’ namespace concept with the addition of RBAC etc.
Service Stable endpoint that forwards requests to (transient) containers Stable endpoint that forwards requests to (transient) pods Stable endpoint that forwards requests to (transient) pods

Understanding Docker – A Tour of Logical Volume Management

A Tour of Logical Volume Management

If you’re using Docker on a RedHat or CentOS based distribution you may have come across logical volume management (LVM). This are used to manage the disk space Docker containers take up and have a number of interesting features. LVM is an abstraction of physical disk to give you more flexibility around allocating and managing the disk space

Learning LVM does not come without challenges, so if you encounter problems with Docker disk usage on this platform it can be useful to take a step back and understand some of the basics of LVM.

To this end I’ve created this introductory video which takes you through the basics of LVM up to creating some thinly-provisioned volumes.

Source available here.

The video was created using ShutIt, an automation tool.

Briefly, it:

  • Assigns a disk to the LVM to manage
  • Sets up a volume group
  • Within this volume group, creates and removes a couple of volumes
  • Creates a thin pool
  • Places two thinly-provisioned volumes within this thin pool (ie these volumes take up space only when written to)
  • Tries to write a file that exceeds one of the volumes’ virtual sizes
  • Extends this volume so that the file can be created
  • Successfully writes the file

There were some odd corner cases I hit that I couldn’t explain while working with LVM – if you are an expert on LVM, please get in touch!

 

Currently co-authoring a book on Docker:

Get 39% off with the code 39miell

dip
https://asciinema.org/a/31978?t=60

Automating Docker Security Validation

Automating Docker Security Checks

Hunting around for ways of validating Docker images from a security perspective, I’ve not seen much documentation on finding ways to do this.

As a result, I put together this video of the best means I’ve found yet: using openscap tooling. You might want to skip to 1:55 when the Vagrant setup is complete and the software is installed.

Source available here.

Briefly, it:

  • Creates a trivial custom check for our security policy (‘is this a CentOS machine?’), which is run using the standard oscap tooling, and passes
  • Runs a RHEL7 container (‘our-rhel-container’)
  • Runs oscap-docker with the same check, which fails (as this is a RHEL7 image)
  • Performs a general CVE check against the container
  • Downloads another policy and runs that against the running container

Using this as a template, you can see how easy it would be to script up a custom policy, run it regularly and perform actions based on the output.

Are you using this or similar tools to manage Docker images? Any tips on extending and improving this gratefully received.

 

Currently co-authoring a book on Docker:

Get 39% off with the code 39miell

dip

The IT Crowd Was Right – What I learned by reading a lot of RFCs

Due to a change of job I’ve recently had to teach myself about networking.

RFCs had always been a bit of a mystery to me but since they came up over and over again when reading about network concepts I thought I’d familiarise myself with them as a whole.

With a quick:

apt-get install rfc-doc

an organised set of RFCs was downloaded and categorised into various folders under /usr/share/doc/RFC. I looked closely at these three:

best-current-practice
for-your-information
standard

and skimmed the rest:

draft-standard, experimental, historic, informational, links, old, proposed-standard, queue, unclassified
Here’s some of the many things I learned by looking through them and reading a good proportion of the active ones:

– There’s a ‘Service Location Protocol’ specification (RFC2608) which anticipates the need for scalable service discovery. Which begs the question: why are we all reinventing the wheel now? There are implementations already written and available (slpd, slptool). Beats me.

– There’s a very handy glossary of internet terms which is still useful (RFC1983), even though written in 1996, is still useful.

– They’re very well written. Really basic things like NFS (RFC1094) and UTF-8 (RFC3629) are explained in a clear and straightforward way.

– There’s nothing quite like dropping the phrase ‘if you read the current RFC on the subject…’ into a meeting.

– The IT Crowd was right – the ‘elders of the internet’ really do exist (RFC1462 – What is the Internet).

Who Governs the Internet?

In many ways the Internet is like a church […] It appoints a council of elders, which has responsibility for the technical management and direction of the Internet.

I’m off to Big Ben to commune with Stephen Hawking.

Understanding Docker – Network Namespaces

Currently co-authoring a book on Docker:

Get 39% off with the code 39miell

dip

 

tl;dr

A video showing how to set up two namespaces to talk to each other based on this excellent tutorial.

Source available here.

What it does

To many developers, networking is a bit of a mystery; it’s the thing to blame when things go wrong.

I’ve been forced to learn about networking more as a result of getting into Docker, and as a result have taught myself some basics.

In this video, (based on this awesome set of tutorials) I take the viewer through the process of setting up network namespaces (a key aspect of Docker’s isolation technology), and use OpenVSwitch (a virtualised switching tool) to act as a bridge between the two networks in the so-called ‘root namespace’. All of this takes place within a VM.

Enjoy!