I work at a very ‘locked-down’ enterprise, where direct access to Docker is effectively verboten.
This, fundamentally, is because access to Docker is effectively giving users root. From Docker’s own pages:
First of all, only trusted users should be allowed to control your Docker daemon.
Most home users get permissions in their account (at least in Linux) by adding themselves to the
docker group, which may as well be root. In Mac, installing Docker also gives you root-like power if you know what you’re doing.
Many Docker platforms (like OpenShift) work around this by putting an API between the user and the Docker socket.
However, for untrusted users this creates a potentially painful dev experience that contrasts badly with their experience at home:
- Push change to git repo
- Wait for OpenShift to detect the change
- Wait for OpenShift to pull the repo
- Wait for OpenShift to build the image
- The last step can take a long time if the build is not cached – which can easily happen when you have lots of build nodes and you miss the cache
vs ‘at home’
- Hit build on your local machine
- See if the build works
What we really want is the capability to build images when you are not root, or privileged in any way.
Thinking about it, you don’t need privileges to create a Docker image. It’s just a bunch of files in a tar file conforming to a spec. But constructing one that conforms to spec is harder than simply building a tar, as you need root-like privileges to do most useful things, like installing rpms or apt packages.
I was excited when I saw Google announced kaniko, which claimed:
…we’re excited to introduce kaniko, an open-source tool for building container images from a Dockerfile even without privileged root access.
Since it doesn’t require any special privileges or permissions, you can run kaniko in a standard Kubernetes cluster, Google Kubernetes Engine, or in any environment that can’t have access to privileges or a Docker daemon.
I took that to mean it could take a Dockerfile and produce a tar file with the image as a non-privileged user on a standard VM. Don’t you?
I also got lots of pings from people across my org asking when they could have it, which meant I had to take time out to look at it.
Pretty quickly I discovered that kaniko does nothing of the sort. You still need access to Docker to build it (which was a WTF moment). Even if you
--force it not to (which you are told not to), you still can’t do anything useful without root.
I’m still not sure why Kaniko exists as a ‘new’ technology when OpenShift already allows users to build images in a controlled way.
After complaining about it on GitHub I got some good advice.
There’s a really useful page here that outlines the state of the art in rootless containers for building, shipping and running.
It’s not for the faint hearted as the range of the technology required is somewhat bewildering, but there’s an enthusiastic mini community of people all trying to make this happen.
A Proof of Concept Build
I managed to get a build of a simple yum install on a
centos:7 base image built as a completely unprivileged user using Vagrant and ShutIt to automate the build.
It shows the build of this Dockerfile as an unprivileged user (
person) who does not have access to the docker socket (Docker does not even need to be installed for the run – it’s only used to build the
proot binary, which could be done elsewhere):
FROM centos:7 RUN yum install -y httpd CMD echo Hello host && sleep infinity
A video of this is available here and the code for this reproducible build is here. The interesting stuff is here.
I can’t claim deep knowledge of the technologies here (so please correct me where I’m wrong/incomplete), but here’s a quick run-down of the technologies used to achieve a useful rootless build.
This allows us to run containers that conform to the OCI spec. Docker (for example) is a superset of the OCI spec of containers.
Although we use ‘runrootless’ to run the containers, runc is still needed to back it (I think – at least I had to install it to get this to work).
Skopeo gives us the ability to take an OCI image and turn it into a Docker one later. It’s also used by orca-build (below) to copy images around while it’s building from Dockerfiles.
umoci modifies container images. It’s also used by orca-build to unpack and repack the image created at each stage. orca-build will
orca-build is a wrapper around runC, and has some support for rootless builds. It uses these technologies:
It also takes Dockerfiles as input (with a subset of Docker commands).
runrootless allows us to run OCI containers as part of each stage of the build, and in turn uses
proot to allow root-like commands.
proot allows you to create a root filesystem (such as is in a Docker container) without having root privileges. I suspected that proot was root by the back door using a setuid flag, but it appears not to be the case (which is good news).
This is required to do standard build tasks like
- User namespaces
These must be switched on in the kernel, so that the build can believe it’s root while actually being an unprivileged user from the point of view of the kernel.
This is currently switched off by default in CentOS, but is easily switched on.
A reproducible VM is available here.
The kaniko work is available here.
Jessie Frazelle’s post on building securely on K8s
Special thanks to @lordcyphar for his great work generally in this area, and specifically helping me get this working.
If you like this post, you might like Learn Git the Hard Way, Learn Bash the Hard Way or Docker in Practice
Get 39% off Docker in Practice with the code: 39miell2
20 thoughts on “Unprivileged Docker Builds – A Proof of Concept”
My take-away is that the solution is to make the workflow through the “Platform Proxy” faster. Missing the cache should not cause a serious delay if you’re mirroring/proxy-caching artifacts inside your enterprise network (Nexus, Artifactory).
True, but that’s easier said than done when you don’t control or have any access to your dependencies.
Reblogged this on josephdung.
I don’t get it why people build such proxy solutions, you can use the auth api endpoint https://docs.docker.com/registry/spec/auth/token/ and limit the interaction with the docker daemon to only the allowed actions
You don’t need a proxy at all, just use the auth API, and limit the interactions with the docker daemon permissions wise: https://docs.docker.com/registry/spec/auth/token/
Isn’t that for the registry?
I looked around your code repository. I see you do something with VirtualBox VMs and Docker, it looks like you just isolate Docker in a VirtualBox VM so that it doesn’t matter if someone has a root access anymore as they can’t escape the VM to access the host system? I mean, that’s definetly one way of doing things. I’d personally would go for KVM-accelerated QEMU rather than VirtualBox, but that’s fine too.
Thanks for looking. I only use a VM to demonstrate the technique in a portable, reproducible and isolated way. The intention would be to run on a standard OS.
The user building the docker image would not need any root privileges at all- that’s the point here. Root is only required to install the software and configure the OS in such a way that the user can build without any privileges. So an OS image could be built by infra for any untrusted users to use and there’s no need to escalate privileges to build images.
And no ‘tricks’ like using setuid involved here either. The only ‘tricks’ are clear from the code, viz user namespaces, which different orgs will have different views on.
Packer? Builds docker images even without a dockerfile