Unprivileged Docker Builds – A Proof of Concept

I work at a very ‘locked-down’ enterprise, where direct access to Docker is effectively verboten.

This, fundamentally, is because access to Docker is effectively giving users root. From Docker’s own pages:

First of all, only trusted users should be allowed to control your Docker daemon.

Most home users get permissions in their account (at least in Linux) by adding themselves to the docker group, which may as well be root. In Mac, installing Docker also gives you root-like power if you know what you’re doing.

Platform Proxies

Many Docker platforms (like OpenShift) work around this by putting an API between the user and the Docker socket.

However, for untrusted users this creates a potentially painful dev experience that contrasts badly with their experience at home:

Push change to git repo
Wait for OpenShift to detect the change
Wait for OpenShift to pull the repo
Wait for OpenShift to build the image
- The last step can take a long time if the build is not cached – which can easily happen when you have lots of build nodes and you miss the cache

vs ‘at home’

Hit build on your local machine
See if the build works

What we really want is the capability to build images when you are not root, or privileged in any way.

Thinking about it, you don’t need privileges to create a Docker image. It’s just a bunch of files in a tar file conforming to a spec. But constructing one that conforms to spec is harder than simply building a tar, as you need root-like privileges to do most useful things, like installing rpms or apt packages.

Kaniko?

I was excited when I saw Google announced kaniko, which claimed:

…we’re excited to introduce kaniko, an open-source tool for building container images from a Dockerfile even without privileged root access.

and

Since it doesn’t require any special privileges or permissions, you can run kaniko in a standard Kubernetes cluster, Google Kubernetes Engine, or in any environment that can’t have access to privileges or a Docker daemon.

I took that to mean it could take a Dockerfile and produce a tar file with the image as a non-privileged user on a standard VM. Don’t you?

I also got lots of pings from people across my org asking when they could have it, which meant I had to take time out to look at it.

Kanikant

Pretty quickly I discovered that kaniko does nothing of the sort. You still need access to Docker to build it (which was a WTF moment). Even if you --force it not to (which you are told not to), you still can’t do anything useful without root.

googledisappointed

I’m still not sure why Kaniko exists as a ‘new’ technology when OpenShift already allows users to build images in a controlled way.

Rootless Containers

After complaining about it on GitHub I got some good advice.

There’s a really useful page here that outlines the state of the art in rootless containers for building, shipping and running.

It’s not for the faint hearted as the range of the technology required is somewhat bewildering, but there’s an enthusiastic mini community of people all trying to make this happen.

A Proof of Concept Build

I managed to get a build of a simple yum install on a centos:7 base image built as a completely unprivileged user using Vagrant and ShutIt to automate the build.

It shows the build of this Dockerfile as an unprivileged user (person) who does not have access to the docker socket (Docker does not even need to be installed for the run – it’s only used to build the proot binary, which could be done elsewhere):

FROM centos:7
RUN yum install -y httpd
CMD echo Hello host && sleep infinity

A video of this is available here and the code for this reproducible build is here. The interesting stuff is here.

Technologies Involved

I can’t claim deep knowledge of the technologies here (so please correct me where I’m wrong/incomplete), but here’s a quick run-down of the technologies used to achieve a useful rootless build.

runC

This allows us to run containers that conform to the OCI spec. Docker (for example) is a superset of the OCI spec of containers.

Although we use ‘runrootless’ to run the containers, runc is still needed to back it (I think – at least I had to install it to get this to work).

skopeo

Skopeo gives us the ability to take an OCI image and turn it into a Docker one later. It’s also used by orca-build (below) to copy images around while it’s building from Dockerfiles.

umoci

umoci modifies container images. It’s also used by orca-build to unpack and repack the image created at each stage. orca-build will

orca-build

orca-build is a wrapper around runC, and has some support for rootless builds. It uses these technologies:

It also takes Dockerfiles as input (with a subset of Docker commands).

runrootless

runrootless allows us to run OCI containers as part of each stage of the build, and in turn uses proot to allow root-like commands.

proot

proot allows you to create a root filesystem (such as is in a Docker container) without having root privileges. I suspected that proot was root by the back door using a setuid flag, but it appears not to be the case (which is good news).

This is required to do standard build tasks like yum or apt commands.

User namespaces

These must be switched on in the kernel, so that the build can believe it’s root while actually being an unprivileged user from the point of view of the kernel.

This is currently switched off by default in CentOS, but is easily switched on.