This is a video going over some of Docker’s new features.
It:
The code for it is here.
My book Docker in Practice
Get 39% off with the code: 39miell
Recently I’ve become responsible for a large project with lots of branches and a complex history I frequently need to dig into. I prefer to work on the command line, so ended up exploring various git log options.
I haven’t found a good write-up of these, so here is my attempt at sharing one with you.
I use this alias everywhere:
alias gitgraph="git log --graph --oneline --all --decorate --topo-order"
In case you’re not familiar with it already, the command to show the history from your current point is:
git log
The output is most-recent commit first, down to the oldest.
Most of the time I don’t care about the author or the date, so in order that I can see more per screen, I use –oneline to only show the commit id and comment per-commit.
git log --oneline
The problem with the above is that you only see a linear series of commits, and get no sense of what was merged in where. To see this aspect of the history use –graph.
git log --oneline --graph
You can see where merges take place, and what commits were merged.
By default you only get the history leading up to the HEAD (ie where you are currently in the git history. Often I want to see all the branches in the history, so I add the –all flag.
git log --graph --oneline --all
That’s great, but I can’t see what branch is where! This is where you use –decorate.
git log --graph --oneline --all --decorate
Now we’re cooking with gas! Each remote or type of branch/tag is shown in a different colour (even stashes!). On my terminal, remotes are in red, HEAD is blue, local branches are in green, stashes in pink.
If you want, you can show the ref name on each line by adding –source, but I usually find this to be overkill:
git log --graph --oneline --all --decorate --source
If you’re looking at the whole history of your project, you may want to see only the significant points of change (ie the lines affected by –decorate above) to eliminate all the intermediary commits. This is perfect for getting an overview of the project as a whole.
git log --graph --oneline --all --decorate --simplify-by-decoration
When viewing the whole history of the project in this way, you might want to re-introduce the Date info with –pretty=
git log --graph --oneline --all --decorate --simplify-by-decoration --pretty='%ar %s %h'
This gives a formatted output, showing (in this case) the relative timestamp (%ar), the commit subject (%s), and the short hash (%h).
You can even see the abstract shape of the git repo!
git log --graph --oneline --all --decorate --simplify-by-decoration --pretty=
The order in which commits are displayed. Although it appears to be the default, it’s worth knowing that –topo-order shows commits ‘bunched together’ by topic, rather than in date order. This leads to easier-to-read graphs in complex projects.
git log --graph --oneline --all --decorate --topo-order
Other options include –date-order and –reverse.
Strangely, you can’t use –graph with –reverse, but you can do this:
git log --graph --oneline --all --decorate --simplify-by-decoration | tac
To show commits from oldest to newest.
(tac is available in the coreutils package – see here for mac)
This is based on work in progress from the second edition of Docker in Practice
Get 39% off with the code: 39miell2
About once every few months I have to set up a reverse tunnel.
I’ve learned the hard way to not read the man page, and just wing it.
After setting one up the other day I looked at the man page to see if it made sense whilst having a picture of its operating still in my mind.
*-R* [bind_address:]port:host:hostport
Specifies that the given port on the remote (server) host is to be forwarded to the given host and port on the local side.
This works by allocating a socket to listen to port on the remote side, and whenever a connection is made to this port, the connec-tion is forwarded over the secure channel, and a connection is made to host port hostport from the local machine.
We all got that, right?
Break it down
OK, maybe I just wasn’t paying close enough attention, so I’m going to read it carefully and take notes while doing so:
At this point I’m basically crying.
Understand it Visually
I hope this helps someone.
Please tweet any corrections or comments to: @ianmiell
My book Docker in Practice
Get 39% off with the code: 39miell
Seventeen months after signing the contract, I hold our book in my hands!
A lot of people have asked me about our experiences writing the book, whether it was ‘worth it’, how much work it was, whether I’d do it again, and should they do one.
I remember googling this before I started, so thought it would be good to pay forward my experiences for the benefit of others.
It’s good to think about why you want to write a book before starting. Reasons include:
None of these are good or bad reasons in themselves.
Vanity and recognition (seeing your name in print, being able to put ‘published author’ on your LinkedIn profile, being able to say ‘I’ve published a book on that’) can be a great motivator to keep going when things get tough, however shallow.
We all need money to live, but it’s no secret that writing a book will not make you rich. That doesn’t mean there aren’t material benefits though! You will get paid if it sells, but it’s not going to be enough in itself to make the hardship worthwhile, and will never be guaranteed. If you want money, market yourself as a consultant and make real money faster.
The secondary benefits are where writing a book really excels. These can include:
In a way, writing a book is like joining a professional club. The editor that persuaded me to write said ‘writing a book will put you into a different professional category’ and that has certainly been my experience.
Conferences can be fun, and you will come into contact with lots of people you would not normally meet. You can also validate your ideas while you write the book.
Whisper it, but writing a book will force you into corners of the technology you might not otherwise look into. In other words, you learn about it! I thought I knew a lot about Docker before I started, but writing a book forced me to up my game. Nothing focusses the mind like knowing that another ‘expert’ will be paid to go over your book with a fine-toothed comb and write brutal feedback!
As another secondary benefit, when you write a book, people ask you questions – a valuable way to gain intelligence about what’s going on in the field!
A book is a great mouthpiece for you. You can take the time to give the benefits of your experience.
For us, we wanted to give the real picture about Docker for engineers. Having spent many months getting it to useful in our business, we felt that a lot of the rhetoric around Docker was misleadingly idealistic, and were keen to make engineers feel OK about compromises they might have to make for their business.
Publishers want to know this before taking you on as an author:
Have a table of contents prepared. Don’t worry if it’s not perfect, the fact that you have something will put you far ahead of the competition, and help convince the publisher you’re serious. They will help you improve it anyway.
This will change a lot as you write, but by knowing the shape of the book in advance – and knowing what you want to say – you will save a lot of time and effort.
I started by giving talks at meetups, which I highly recommend for all sorts of reasons, but having a video of me communicating to my peers ticked a lot of these boxes for my publisher.
Get a good publisher. It’s worth far more than a few percent more on the book’s sales.
A good publisher provides:
That’s quite a list! You can self-publish if you want, and you will make a bigger cut, but you will have to do a lot more work to sell each copy.
Maybe the time will come that I write ‘SLAVE’ on my face, but I can always self-publish at that point and save on marker pen ink.
In fact, I had a look at some self-published technical books in preparation for this article, and could very quickly see flaws not in content but definitely in copy, approach, style and design. All things the Manning team saved me from.
I have nothing but good things to say about the team at Manning, who had a very efficient team which supported and pushed us to make a better (and a better-selling) book. And we learned a lot about the publishing industry through writing it.
For example, about 20% of the writing time was spent getting the first chapter right. We went over it again and again until we’d learned the lessons we needed to. After that, the subsequent chapters came out pretty easily.
It’s no surprise that writing book is time-consuming, but there are some subtleties beyond this to consider.
Simply taking 30 minutes every day to ‘work on the book’ is not enough. That will work for writing linking snippets or bits of proofing or testing, but will need stretches of many hours to do the real meat of the work. I would take myself off to the libraries for many stretches to focus on sections of the book that required real thought.
For example, I spent more time than I care to relate trying to get Kubernetes set up at home when it was still in its infancy. In retrospect this was a massive waste of time! But quite valuable in other ways – I learned a lot about the product, and even got into contact with the lead Kubernetes developer.
I was lucky in that I had two months between jobs, so I wrote and explored London, which was a fantastic life experience and a lot of fun. If I were to write a book again I would probably quit work and devote myself to it full-time. It’s less profitable, but getting the focus full-time would make life a lot easier and fun.
I would seriously recommend getting a co-author. The benefits are numerous:
But not at any cost. I was lucky with my co-author – the work divided up nicely between us, and our skills complemented each other nicely. If you don’t trust someone, don’t sign up with them, as the process can get stressful if you let it.
I would have to think very carefully about going through this process again on my own.
Sounds obvious, but you will need to be able to write. I was lucky that I’d had some professional writing experience, but when you write in a commercial niche there’s more to learn about what works and what doesn’t.
If you can’t write already, be prepared to listen and learn. A good editor will give robust feedback, and tell you clearly where you need to improve.
And get good at doing diagrams. They don’t need to be polished, but diagrams are increasingly important in successful technical books.
When you see tweets like this it makes it all that effort worthwhile.
Although publishers provide marketing for you, you should be willing to work on this side of things as well. You should know what topics are of interest to people, and the marketing team will be very grateful if you feed them useful content.
Feel free to contact me if you want more advice or have any questions!
This post is about my book Docker in Practice
Get 39% off with the code: 39miell
Following on from my earlier post on practising grep, this post shows two tutorials you can run on the command line to gain familiarity and get practice in with git bisect and rebase in a realistic and safe way.
They use Docker and ShutIt to help manage state in safe and lightweight way, while guiding you through the process.
Please tweet feedback to @ianmiell
More info here, the code is here
More info here, the code is here
If you haven’t seen it, this is a very accurate and smart Downfall satire on Docker’s ecosystem, technology, and culture.
It’s so good that I thought it would be instructive to annotate it so that the state of the art and some technical details of Docker could be better explained.
Tweet me @ianmiell if you spot a problem or want to suggest an improvement.
Henchman: We pushed the images to DockerHub, then used docker-compose to deploy to the cluster
Docker Hub is the public repository of Docker containers,
analagous to GitHub.
docker-compose is a tool for managing multiple
containers as a unit on a single machine.
You’d more likely use docker-swarm to deploy to a cluster
Henchman: We mounted data volumes on these nodes, and linked the app container here. Finally we updated the dns records.
A data volume is the persistent store of data for containers.
Since containers are generally ephemeral, persistent data
is ‘mounted’ to containers on startup.
Hitler: So we’re running 20 containers on every node now. When can we get rid of the excess servers?
It is a promising aspect of Docker that it can reduce your
physical server footprint through ‘max-packing’ containers
on physical tin through reduced resource claims and not
needing to run multiple kernels (the so called ‘Hypervisor tax‘).
Henchman: Mein Fuerer, the kernel… A third party container caused a kernel panic.
Docker containers talk to the Linux API. If they can
cause a container panic outside of Docker, then they will
inside Docker also. Docker allows some safeguards
against this, for example by reducing capabilities
and/or system calls the container can use.
Henchman: We’ve lost 70% of the cluster and the data volumes
Presumably the Kernel panic caused this loss.
Presumably also Hitler blames Docker users’ encouragement
of free container downloading as the cause (see below).
Hitler: If you never used Docker in production, leave the room now
Figures are hard to come by, but Docker use in production
heavily lags development and test. This is to be expected,
since most new technologies bubble up through developers to
production.
Hitler: What were you thinking? Who the hell uses public containers from DockerHub? For all you know they were made by Russian hackers!
Container security is a hot topic.
Hitler: You might as well use `curl | sudo bash`!
A convenient means of distributing software that effectively hands
over control of your machine to the internet.
Hitler: You think anything in public repos is secure because it’s OSS? You’re a bunch of node.js hipsters that just HAVE to install everything you read on Hacker News!
Guilty (but not the node.js bit).
Henchman: But Docker allows us to run our application anywhere!
A paraphrase of Docker’s slogan. ‘Build, ship, and run any app anywhere.’
Hitler: You use a VM just to run Docker on your laptop!
Many users can’t run Docker directly on their machines,
for example if they use Windows or OSX. I use
Docker on an Ubuntu VM running on a Mac.
I’m not sure what Hitler would make of this.
Henchman: Main Fuerer, docker-machine uses a lightweight VM!
Docker uses a lightweight Linux distro to run on OSX/Windows.
Hitler: Do you hear yourself? Why do we need docker if we’re running a VM? A container inside a container!!!
In this context, a VM can be used for isolation,
so is considered a container also. A docker advocate
would argue that the image is lightweight and easier
to deploy than a VM. Or as I like to say, Docker doesn’t
give you anything a VM can’t, but a computer gives you
nothing nothing an abacus can’t – user experience is key.
Hitler: You archived a whole Linux O/S then used CoW storage because it’s too big.
By ‘Linux O/S’ Hitler here means an
operating system’s filesystem, which makes up most
of a Docker image. CoW (copy on write) storage is a
feature of Docker where changes to the filesystem
are copied on write to make a new ‘layer‘ ready
for committing as a new image. Images are made
up of these layers, which can be shared between
containers, reducing disk usage. Hitler’s point here
is that the images contain a lot of data, which can be wasteful.
Hitler: Just so you can deploy a 10MB go binary!
Docker is written in Go, a fashionable language. One
of Go’s features is that it generates portable
binaries that can be run across different distributions
with little difficulty. Hitler is making the point that
if that Go binary is portable, why bother with Docker at all?
Hitler: Don’t even talk to me about resource constraints. All that cgroups magic and it still can’t stop a simple fork bomb!
CGroups is a technology used by Docker and others to attempt to
ensure fair (or guaranteed) resource allocation. It can be tricky to
learn. Fork bomb attacks have been known to work on Docker,
but work has been done on this recently.
Hitler: And if the database needs all the resources on the server how exactly will Docker allow you to run more programs on it!? Before Docker I just picked the right size VMs.
Docker is not magic. If you need the tin for your application,
it won’t help you get more resources.
Hitler: Suddenly people talk to me about datacenter efficiency and “hyperconvergence”. Everybody thinks they’re Google!
Far too many organisations act like they are
running at Google scale when they are not.
Hitler: You don’t even run your own machines anymore! People run on GCE, in VM instances that run in Linux containers on Borg!
Google Compute Engine is Google’s alternative to Amazon
Web Services. They run VMs within Linux containers that
themselves run Docker, which presumably Hitler thinks is
laughable, but is there to provide greater levels of
security, and likely because Google is not short of
compute! Borg is Google’s cluster management
software, on which Kubernetes is based.
Hitler: People even think Docker is configuration management. They think Docker solves everything!
If Docker is anything, it’s package management.
You might use Dockerfiles for primitive configuration
management, but you can use traditional
CM tools like Chef and Puppet to provision your images.
Hitler: Even Microsoft has containers now. I’m moving everyone to Windows!
The Windows picture is quite complicated. You can:
– Run Docker within a VM running on Windows (see above)
– Run a Windows container (not widely available yet)
that implements the Docker API. This will talk to the Windows
OS API (I assume) rather than the Linux Kernel API
so the images built will not run across the systems.
– Run bash in Windows natively. See below.
Henchwoman: Don’t cry, you can run bash on windows 10 now.
You can. In a Windows Linux ‘subsystem’. This is not a VM technology.
Hitler: Docker is supposed to have better performance yet that userland proxy is slower than a 28.8k modem and for what, just bind on port 0.
A userland proxy is one written in software and outside
the kernel. In-kernel proxies are much faster.
Binding on port 0 gets any available port from the OS.
Docker does something similar by default.
Docker performance is not better than natively-run software,
but in some cases is arguably better than VMs.
Hitler: Even enterprises want to run Docker now and they still have Red Hat 5 installed.
This happens.
RedHat is an enterprise-supported implementation
of Linux. RedHat5 was released in 2007.
Hitler: You idiots that Docker will help your application scale.
It won’t. It can allow you to run more instances
of your application, which is not the same thing.
Hitler: Use Openstack for all I care.
Openstack is an open-source cloud technology, which is
powerful but costly to manage, and somewhat out of favour now.
Author is currently working on the second edition of Docker in Practice
Get 39% off with the code: 39miell2
The idea behind this tool is to reinforce Linux command-line knowledge through repeated and guided practice in a safe (Docker) environment.
Here’s a video of it in action:
It uses an automation tool called ShutIt to manage inputs and outputs in a neat programmable way.
Source is here.
I welcome suggestions for other scales to produce!
This is an update to a previous post where I went over some Kubernetes basics and showed how to get it working using Docker.
It’s worth reading that before this if you need a primer.
I’ve updated the code so you can play along more easily, and tinker using Vagrant and Virtualbox.
With three commands you can watch Kubernetes play out in front of you:
sudo pip install shutit
git clone --recursive <this repo>
./run.shHere’s a video:
If you want to play along, run:
./run.sh --interactive 2
and follow the instructions.
Or just hit CTRL-C to get a terminal during the run.
This post is based on material from my book Docker in Practice
Get 39% off with the code: 39miell2
Let’s say you have a server that has been lovingly hand-crafted that you want to containerize.
Figuring out exactly what software is required on there and what config files need adjustment would be quite a task, but fortunately blueprint exists as a solution to that.
What I’ve done here is automate that process down to a few simple steps. Here’s how it works:
You kick off a ShutIt script (as root) that automates the bash interactions required to get a blueprint copy of your server, then this in turn kicks off another ShutIt script which creates a Docker container that provisions the container with the right stuff, then commits it. Got it? Don’t worry, it’s automated and only a few lines of bash.
There are therefore 3 main steps to getting into your container:
– Install ShutIt on the server
– Run the ‘copyserver’ ShutIt script
– Run your copyserver Docker image as a container
Install ShutIt as root:
sudo su -
pip install shutit
The pre-requisites are python-pip, git and docker. The exact names of these in your package manager may vary slightly (eg docker-io or docker.io) depending on your distro.
You may need to make sure the docker server is running too, eg with ‘systemctl start docker’ or ‘service docker start’.
Check out the copyserver script:
git clone https://github.com/ianmiell/shutit_copyserver.git
If you like this, you might like one of my books:
Learn Bash the Hard Way
Learn Git the Hard Way
Learn Terraform the Hard Way
Run the copy_server script:
cd shutit_copyserver/bin ./copy_server.sh
There is a prompt to ask what docker base image you want to use. Make sure you use one as close to the original server as possible.
Run the built image:
docker run -ti [username]/copyserver /bin/bash
You are now in a practical facsimile of your server within a docker container!
This article continues on from the previous two posts outlining a method of provisioning Jenkins instances on demand programatically using docker-compose.
In this post we take this one step further by demonstrating how a Docker container can dynamically add itself as a client to the Jenkins server.
This updated diagram shows the architecture at part III
The ‘Jenkins-Swarm Docker Slave’ is new to this example. It is similar to the ‘Jenkins SSH Slave’ except that it connects itself to the Jenkins server as a slave running as a Docker container. Using this as a template you can dynamically add multiple clients to the Jenkins server under the ‘swarm’ tag.
Note: Do not confuse ‘Jenkins Swarm’ with ‘Docker Swarm’.
They are two different things: Jenkins Swarm allows clients to
dynamically attach themselves to a server and run jobs, while
Docker Swarm is a clustering solution for Docker servers.
In addition, these new plugins have been added:
swarm – allow dynamic jenkins clients to be added via port 50000
backup – backs up configuration on demand (basic configuration set up by scripts)
jenkinslint – gives advice on your jenkins setup
build-timeout – allow a build timeout
docker-build-publish – build and publish Docker projects to the Docker Hub
greenballs – green balls for success, not blue!
This is now baked into v3.0 of the git repo.
Skip to the end for a demo of a simple job running on a swarm node.