Summary
Following on from my previous post setting up an OpenShift cluster in Vagrant, this post discusses migrating an etcd cluster within a live OpenShift instance to newer servers.
Moving a standalone etcd cluster is relatively straightforward, but when it’s part of an OpenShift cluster — and especially one that’s live and operational — it is a little more involved.
The ordering of actions is important and there are several aspects to consider when planning such a move:
- Config management preparation
- Stopping the cluster
- Creation and distribution of certificates
- Data migration
- Update of OpenShift config
- Update of config management
Here we are using Ansible to provision and maintain the environment.
You can also use Chef to manage your OpenShift cluster.
Code
The code for this is here:
Video
Here’s a video of the upgrade process:
Steps
VM Setup
This section of the code sets up the VMs using Vagrant.
Cluster Setup
The next section sets up the OpenShift cluster. It:
- sets up ssh access across all the hosts
- writes the ansible hosts config file
- triggers the ansible playbook
Take a Backup
Take a backup of etcd on all three nodes, just in case.
Stop the Cluster
Generate New Certs
For each new node, run the commands to generate the certs for the new nodes, and copy to the codes.
Add etcd Nodes One-By-One
Again for each node:
- add the new node to the etcd cluster
- go to the node
- install etcd
- extract the certificates
- update the etcd config
- restart etcd
NOTE: If you have a lot of data in your cluster, you will want to give the new node ample time to receive the data from the other nodes. In this trivial example, there is little data to transfer. Alternatively, you can copy over the data from one of the original nodes.
Drop the Old Members
Now drop the old members from the cluster and remove etcd from those hosts:
Update the Master Config and Bring the OpenShift Cluster Back Up
The /etc/origin/master/master-config.yaml file needs to updated to reflect the new etcd cluster before bringing back the OpenShift cluster.
Update Config Manager and Re-Run
Learn More
My book Docker in Practice:

Get 39% off with the code: 39miell
4 thoughts on “Migrating an OpenShift etcd Cluster”