Goodbye Docker: Purging is Such Sweet Sorrow

After 6 years, I removed Docker from all my home servers.

apt purge -y docker-ce

Why?

This was triggered by a recurring incident I faced where the Docker daemon was using 100% CPU on multiple cores that made the host effectively unusable.

This had happened a few times before, and was likely due to a script that had got out of hand starting up too many containers. I’d never really got to the bottom of it, as I had to run a command to kill off all the containers and restart the daemon. This time, the daemon wouldn’t restart without a kill -9, so I figured enough was enough.

Anyway, I didn’t necessarily blame Docker for it, but it did add force to an argument I’d heard before:

Why does Docker need a daemon at all?

Podman, Skopeo, and Buildah

These three tools are an effort mostly pushed by RedHat that do everything I need Docker to do. They don’t require a daemon or access to a group with root privileges.

Podman

Podman replaces the Docker command for most of its sub-commands (run, push, pull etc). Because it doesn’t need a daemon, and uses user namespacing to simulate root in the container, there’s no need to attach to a socket with root privileges, which was a long-standing concern with Docker.

Buildah

Buildah builds OCI images. Confusingly, podman build can also be used to build Docker images also, but it’s incredibly slow and used up a lot of disk space by using the vfs storage driver by default. buildah bud (‘build using Dockerfile’) was much faster for me, and uses the overlay storage driver.

The user namespacing allowing rootless builds was the other killer feature that made me want to move. I wrote a piece about trying to get rootless builds going last year, and now it comes out of the box with /etc/subuid and /etc/subgid set up for you, on Ubuntu at least.

Skopeo

Skopeo is a tool that allows you to work with Docker and OCI images by pushing, pulling, and copying images.

The code for these three are open source and available here:

Podman

Buildah

Skopeo

Steps to Move

Installing these tools on Ubuntu was a lot easier than it was 6 months ago.

I did seem to have to install runc independently of those instructions. Not sure why it wasn’t a pre-existing dependency.

First, I replaced all instances of docker in my cron and CI jobs with podman. That was relatively easy as it’s all in my Ansible scripts, and anything else was a quick search through my GitHub repos.

Once that was bedded in, I could see if anything else was calling docker by using sysdig to catch any references to it:

sysdig | grep -w docker

This may slow down your system considerably if you’re performance-sensitive.

Once happy that nothing was trying to run docker, I could run:

apt remove -y docker-ce

I didn’t actually purge in case there was some config I needed.

Once everything was deemed stable, the final cleanup could take place:

  • Remove any left-over sources in /etc/apt/* that point to Docker apt repos
  • Remove the docker group from the system with delgroup docker
  • Remove any left-over files in etc/docker/*, /etc/default/docker and /var/lib/docker

A few people asked what I did about Docker Compose, but I don’t use it, so that wasn’t an issue for me.

Edit: there exists a podman-compose project,
but it’s not considered mature.

Differences?

So far, and aside from the ‘no daemon’ and ‘no sudo access required’, I haven’t noticed many differences.

Builds are local to my user (in ~/.local/containers) rather than global (in /var/lib/docker), in keeping with the general philosophy of these tools as user-oriented rather than daemon-oriented. But since my home servers have only one user using Docker, that wasn’t much of an issue.

The other big difference I noticed was that podman pull downloads get all layers in parallel, in contrast to Docker’s. I don’t know if this causes problems if too many images are being pulled at once, but that wasn’t a concern for me.


If you like this, you might like one of my books:
Learn Bash the Hard Way

Learn Git the Hard Way
Learn Terraform the Hard Way

LearnGitBashandTerraformtheHardWay

Get 39% off Docker in Practice with the code: 39miell2

37 thoughts on “Goodbye Docker: Purging is Such Sweet Sorrow

  1. You said that some of the containers were consuming 100% of CPU cores, but you said nothing about trying to limit the resources. If your containers consume too much, perhaps you did not configure them of you docker right?

  2. I’ve switched to podman and buildah too. Now our CI system is happy to build all the stuff faster and without painfull workarounds.
    Docker just has too many bugs lasting forever.
    I didn’t even change docker calls for that as podman has podman-docker wrapper which allows you to call docker command to execute actual podman.
    There was some refactoring in dockerfiles as buildah had behaven more restrictive way.

  3. Unless you want docker instances to run at specific time cron shouldnt be use if it is ob restart of system you should tell docker to start a container in bot that is why docker requires a daemon also for control software to run independant of any user as a service it also makes it a tad bit more secure during shutdowns

  4. Set up live reload, the daemon and containers are then independent. The daemon can be restarted without affecting containers.

    I’ve rarely seen the daemon driving up cpu usage but you can use tools like this to automatically shoot and restart it. Likely a daemon using all the cpu also isn’t being responsive so this script will restart it https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/health-monitor.sh

    Obviously youve done the legwork to switch off docker but these are easier options for people that continue to use docker. It is the most reliable solution today

  5. I get where you’re coming from with this but can’t help wondering why the modern webdev solutions to problematic software components is always to replace them with multiple new solutions that add even more complexity to an already overly complex stack.

    1. Perhaps for some people it is less complex? I find the approach of one tool to do it all to be more complex than a set of tools where each focuses on doing one thing (commonly called the Unix approach). I can understand each simple tool in its entirety knowing it has a small bounded scope. I can then compose those tools is ways the original designers may not have thought of.

      I like building blocks. Some like all in one. Luckily we have both now and thanks to the author of this article, people who prefer the first approach now have a guide to getting started.

  6. We got this issue with high CPU usage by docker daemon, we ended up manually upgrade to docker ce 18.9 version and the issue is gone

  7. Advertising your Docker book below a post where you are talking about how you moved away from it.

    p.s. Such sweet thunder

    1. What say that and not provide a link or at least a reason about your claims?

      Obvious troll is obvious…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s