I work at a very ‘locked-down’ enterprise, where direct access to Docker is effectively verboten.
This, fundamentally, is because access to Docker is effectively giving users root. From Docker’s own pages:
First of all, only trusted users should be allowed to control your Docker daemon.
Most home users get permissions in their account (at least in Linux) by adding themselves to the
docker group, which may as well be root. In Mac, installing Docker also gives you root-like power if you know what you’re doing.
Many Docker platforms (like OpenShift) work around this by putting an API between the user and the Docker socket.
However, for untrusted users this creates a potentially painful dev experience that contrasts badly with their experience at home:
- Push change to git repo
- Wait for OpenShift to detect the change
- Wait for OpenShift to pull the repo
- Wait for OpenShift to build the image
- The last step can take a long time if the build is not cached – which can easily happen when you have lots of build nodes and you miss the cache
vs ‘at home’
- Hit build on your local machine
- See if the build works
What we really want is the capability to build images when you are not root, or privileged in any way.
Thinking about it, you don’t need privileges to create a Docker image. It’s just a bunch of files in a tar file conforming to a spec. But constructing one that conforms to spec is harder than simply building a tar, as you need root-like privileges to do most useful things, like installing rpms or apt packages.
I was excited when I saw Google announced kaniko, which claimed:
…we’re excited to introduce kaniko, an open-source tool for building container images from a Dockerfile even without privileged root access.
Since it doesn’t require any special privileges or permissions, you can run kaniko in a standard Kubernetes cluster, Google Kubernetes Engine, or in any environment that can’t have access to privileges or a Docker daemon.
I took that to mean it could take a Dockerfile and produce a tar file with the image as a non-privileged user on a standard VM. Don’t you?
I also got lots of pings from people across my org asking when they could have it, which meant I had to take time out to look at it.
Pretty quickly I discovered that kaniko does nothing of the sort. You still need access to Docker to build it (which was a WTF moment). Even if you
--force it not to (which you are told not to), you still can’t do anything useful without root.
I’m still not sure why Kaniko exists as a ‘new’ technology when OpenShift already allows users to build images in a controlled way.
After complaining about it on GitHub I got some good advice.
There’s a really useful page here that outlines the state of the art in rootless containers for building, shipping and running.
It’s not for the faint hearted as the range of the technology required is somewhat bewildering, but there’s an enthusiastic mini community of people all trying to make this happen.
A Proof of Concept Build
It shows the build of this Dockerfile as an unprivileged user (
person) who does not have access to the docker socket (Docker does not even need to be installed for the run – it’s only used to build the
proot binary, which could be done elsewhere):
FROM centos:7 RUN yum install -y httpd CMD echo Hello host && sleep infinity
I can’t claim deep knowledge of the technologies here (so please correct me where I’m wrong/incomplete), but here’s a quick run-down of the technologies used to achieve a useful rootless build.
This allows us to run containers that conform to the OCI spec. Docker (for example) is a superset of the OCI spec of containers.
Although we use ‘runrootless’ to run the containers, runc is still needed to back it (I think – at least I had to install it to get this to work).
Skopeo gives us the ability to take an OCI image and turn it into a Docker one later. It’s also used by orca-build (below) to copy images around while it’s building from Dockerfiles.
umoci modifies container images. It’s also used by orca-build to unpack and repack the image created at each stage. orca-build will
orca-build is a wrapper around runC, and has some support for rootless builds. It uses these technologies:
It also takes Dockerfiles as input (with a subset of Docker commands).
runrootless allows us to run OCI containers as part of each stage of the build, and in turn uses
proot to allow root-like commands.
proot allows you to create a root filesystem (such as is in a Docker container) without having root privileges. I suspected that proot was root by the back door using a setuid flag, but it appears not to be the case (which is good news).
This is required to do standard build tasks like
- User namespaces
These must be switched on in the kernel, so that the build can believe it’s root while actually being an unprivileged user from the point of view of the kernel.
This is currently switched off by default in CentOS, but is easily switched on.
A reproducible VM is available here.
The kaniko work is available here.
Special thanks to @lordcyphar for his great work generally in this area, and specifically helping me get this working.