A ‘Hello World’ GitOps Example Walkthrough

Intro

This post walks through a ‘hello world’ GitOps example I use to demonstrate key GitOps principles.

If you’re not aware, GitOps is a term coined in 2017 to encapsulate certain engineering principles that were becoming more common with the advent of recent tooling in the area of software deployment and maintenace.

If you want to know more about the background and significance of GitOps, I wrote an ebook on the subject, available for download here from my company. One of the more fun bits of writing that book was creating this diagram, which seeks to show the historical antecedents to the latest GItOps tooling, divided on the three principles of declarative code, source control, and distributed control loop systems.

This post is more a detailed breakdown of one implementation of a trivial application. It uses the following technologies and tools:

• Docker
• Kubernetes
• GitHub
• GitHub Actions
• Shell
• Google Kubernetes Engine
• Terraform

Overview

The example consists of four repositories:

• https://github.com/ianmiell/gitops-example

• https://github.com/ianmiell/gitops-example-app

• https://github.com/ianmiell/gitops-example-deploy

• https://github.com/ianmiell/gitops-example-infra

It should be viewed in conjunction with this diagram to get an overview of what’s going on in the example. I’ll be referring to the steps from 0 to 5 in some detail below:

There are three ‘actors’ in this example: a developer (Dev), an operations engineer (Ops), and an Infrastructure engineer (Infra). The Dev is responsible for the application code, the Ops is responsible for deployment, and the Infra is responsible for the platform on which the deployment runs.

The repository structure reflects this separation of concerns. In reality, all roles could be fulfilled by the same person, or there could be even more separation of duties.

Also, the code need not be separated in this way. In theory, just one repository could be used for all four purposes. I discuss these kind of ‘GitOps Decisions’ in my linked post.

---

If you like this, you might like one of my books:
Learn Bash the Hard Way
Learn Git the Hard Way
Learn Terraform the Hard Way

---

The Steps

Here’s an overview of the steps outlined below:

• A – Pre-Requisites
• B – Fork The Repositories
• C – Create The Infrastructure
• D – Set Up Secrets And Keys
  • D1 – Docker Registry Login Secret Setup
  • D2 – Set Up Repository Access Token
  • D3 – Install And Set Up FluxCD
• E – Build And Run Your Application

A – Pre-Requisites

You will need:

• A GitHub account
• A Google Cloud account
• A Docker registry account (eg docker.com)
• kubectl installed to your host. See here
• fluxctl installed to your host. See here
• gcloud installed to your host. See here

B – Fork the Repositories

Fork these three repositories to your own GitHub account:

• https://github.com/ianmiell/gitops-example-app

• https://github.com/ianmiell/gitops-example-deploy

• https://github.com/ianmiell/gitops-example-infra

C – Create the Infrastructure

This step uses the infra repository to create a Kubernetes cluster on which your workload will run, with its configuration being stored in code.

This repository contains nothing in the main branch, and a choice of branches depending on the cloud provider you want to choose.

The best-tested branch is the Google Cloud Provider (gcp) branch, which we cover here.

The code itself consists of four terraform files:

• connections.tf
  • defines the connection to GCP
• kubernetes.tf
  • defines the configuration of a Kubernetes cluster
• output.tf
  • defines the output of the terraform module
• vars.tf
  • variable definitions for the module

To set this up for your own purposes:

• Check out the gcp branch of your fork of the code
• Set up a Google Cloud account and project
• Log into Google Cloud on the command line:
  • gcloud auth login
  • Update components in case they have updated since gcloud install:
    • gcloud components update
• Set the project name
  • gcloud config set project <GCP PROJECT NAME>
• Enable the GCP container APIs
  • gcloud services enable container.googleapis.com
• Add a terraform.tfvars file that sets the following items:
  • cluster_name
    • Name you give your cluster
  • linux_admin_password
    • Password for the hosts in your cluster
  • gcp_project_name
    • The ID of your Google Cloud project
  • gcp_project_region
    • The region in which the cluster should be located, default is us-west-1
  • node_locations
    • Zones in which nodes should be placed, default is ["us-west1-b","us-west1-c"]
  • cluster_cp_location
    • Zone for control plane, default is us-west1-a
• Run terraform init
• Run terraform plan
• Run terraform apply
• Get kubectl credentials from Google, eg:
  • gcloud container clusters get-credentials <CLUSTER NAME> --zone <CLUSTER CP LOCATION>
• Check you have access by running kubectl cluster-info
• Create the gitops-example namespace
  • kubectl create namespace gitops-example

If all goes to plan, you have set up a kubernetes cluster on which you can run your workload, and you are ready to install FluxCD. But before you do that, you need to set up the secrets required across the repositories to make all the repos and deployments work together.

D – Set Up Secrets And Keys

In order to co-ordinate the various steps in the GitOps workflow, you have to set up three sets of secrets in the GitHub repositories. This is to allow:

1. The Kubernetes cluster to log into the Docker repository you want to pull your image from
2. The github-example-app‘s repository action to update the image identifier in the github-example-deploy repository
3. Allow fluxcd to access the gitops-example-deploy GitHub repository from the Kubernetes cluster

D1. Docker Registry Login Secret Setup

To do this you create two secrets in the gitops-example-app repository at the link:

https://github.com/<YOUR GITHUB USERNAME>/gitops-example-app/settings/secrets/actions

• DOCKER_USER
  • Contains your Docker registry username
• DOCKER_PASSWORD
  • Contains your Docker registry password

Next, you set up your Kubernetes cluster so it has these credentials.

• Run this command, replacing the variables with your values:

kubectl create -n gitops-example secret docker-registry regcred --docker-server=docker.io --docker-username=$DOCKER_USER --docker-password=$DOCKER_PASSWORD --docker-email=$DOCKER_EMAIL

D2. Set up Repository Access Token

To do this you first create a personal access token in GitHub.

• You can do this by visiting this link. Once there, generate a token called EXAMPLE_GITOPS_DEPLOY_TRIGGER.

• Give the token all rights on repo, so it can read and write to private repositories.

• Copy that token value into a secret with the same name (EXAMPLE_GITOPS_DEPLOY_TRIGGER) in your gitops-example-app at:

https://github.com/<YOUR GITHUB USERNAME>/gitops-example-app/settings/secrets/actions

D3 – Install And Set Up FluxCD

Finally, you set up flux in your Kubernetes cluster, so it can read and write back to the gitops-example-deploy repository.

• The most up-to-date FluxCD deployment instructions can be found here, but this is what I run on GCP to set up FluxCD on my cluster:

kubectl create clusterrolebinding "cluster-admin-$(whoami)" --clusterrole=cluster-admin --user="$(gcloud config get-value core/account)"
kubectl create ns flux
fluxctl install --git-branch main --git-user=<YOUR GITHUB USERNAME> --git-email=<YOUR GITHUB EMAIL> --git-url=git@github.com:<YOUR GITHUB USERNAME>/gitops-example-deploy --git-path=namespaces,workloads --namespace=flux | kubectl apply -f -

• When the installation is complete, this command will return a key generated by FluxCD on the cluster:

fluxctl identity --k8s-fwd-ns flux

• You need to take this key, and place it in the gitops-example-deploy repository at this link:

https://github.com/<YOUR GITHUB USERNAME>/gitops-example-deploy/settings/keys/new

• Call the key flux
• Tick the ‘write access’ option
• Click ‘Add Key’

You have now set up all the secrets that need setting up to make the flow work.

You will now make a change and will follow the links in the steps as the application builds and deploys without intervention from you.

E – Build And Run Your Application

To deploy your application, all you need to do is make a change to the application in your gitops-example-app repository.

• Step 1a, 2 and 3

Go to:

https://github.com/<YOUR GITHUB USERNAME>/gitops-example-app/blob/main/Dockerfile

and edit the file, changing the contents of the echo command to whatever you like, and commit the change, pushing to the repository.

This push (step 1a above) triggers the Docker login, build and push via a GitHub action (steps 2 and 3), which are specified in code here:

https://github.com/ianmiell/gitops-example-app/blob/main/.github/workflows/main.yaml#L13-L24

This action uses a couple of docker actions (docker/login-action and docker/push-action) to commit and push the new image with a tag of the github SHA value of the commit. The SHA value is given to you as a variable by GitHub Actions (github.sha) within the action’s run. You also use the DOCKER secrets set up earlier. Here’s a snippet:

    - name: Log in to Docker Hub
      uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9
      with:
        username: ${{secrets.DOCKER_USER}}
        password: ${{secrets.DOCKER_PASSWORD}}
    - name: Build and push Docker image
      id: docker_build
      uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc
      with:
        context: .
        push: true
        tags: ${{secrets.DOCKER_USER}}/gitops-example-app:${{ github.sha }}

• Step 4

Once the image is pushed to the Docker repository, another action is called which triggers another action that updates the gitops-example-deploy Git repository (step 4 above)

    - name: Repository Dispatch
      uses: peter-evans/repository-dispatch@v1
      with:
        token: ${{ secrets.EXAMPLE_GITOPS_DEPLOY_TRIGGER }}
        repository: <YOUR GITHUB USERNAME>/gitops-example-deploy
        event-type: gitops-example-app-trigger
        client-payload: '{"ref": "${{ github.ref }}", "sha": "${{ github.sha }}"}'

It uses the Personal Access Token secrets EXAMPLE_GITOPS_DEPLOY_TRIGGER created earlier to give the action the rights to update the repository specified. It also passes in an event-type value (gitops-example-app-trigger) so that the action on the other repository knows what to do. Finally, it passes in a client-payload, which contains two variables: the github.ref and the github.sha variables made available to us by the GitHub Action.

This configuration passes all the information needed by the action specified in the gitops-example-deploy repository to update its deployment configuration.

The other side of step 4 is the ‘receiving’ GitHub Action code here:

https://github.com/ianmiell/gitops-example-deploy/blob/main/.github/workflows/main.yaml

Among the first lines are these:

on:
  repository_dispatch:
    types: gitops-example-app-trigger

Which tell the action that it should be run only on a repository dispatch, when the event type is called gitops-example-app-trigger. Since this is what we did on the push to the gitops-example-app action above, this should be the action that’s triggered on this gitops-example-deploy repository.

The first thing this action does is check out and update the code:

      - name: Check Out The Repository
        uses: actions/checkout@v2
      - name: Update Version In Checked-Out Code
        if: ${{ github.event.client_payload.sha }}
        run: |
          sed -i "s@\(.*image:\).*@\1 docker.io/${{secrets.DOCKER_USER}}/gitops-example-app:${{ github.event.client_payload.sha }}@" ${GITHUB_WORKSPACE}/workloads/webserver.yaml

If a sha value was passed in with the client payload part of the github event, then a sed is performed, which updates the deployment code. The workloads/webserver.yaml Kubernetes specification code is updated by the sed command to reflect the new tag of the Docker image we built and pushed.

Once the code has been updated within the action, you commit and push using the stefanzweifel/git-auto-commit-action action:

      - name: Commit The New Image Reference
        uses: stefanzweifel/git-auto-commit-action@v4
        if: ${{ github.event.client_payload.sha }}
        with:
          commit_message: Deploy new image ${{ github.event.client_payload.sha }}
          branch: main
          commit_options: '--no-verify --signoff'
          repository: .
          commit_user_name: Example GitOps Bot
          commit_user_email: <AN EMAIL ADDRESS FOR THE COMMIT>
          commit_author: <A NAME FOR THE COMMITTER> << AN EMAIL ADDRESS FOR THE COMMITTER>>

• Step 5

Now the deployment configuration has been updated, we now wait for FluxCD to notice the change in the Kubernetes deployment configuration. After a few minutes, the Flux controller will notice that the main branch of the gitops-example-deploy repository has changed, and try the apply the yaml configuration in that repository to the Kubernetes cluster. This will update the workload.

If you port-forward to the application’s service, and hit it using curl or your browser, you should see that the application’s output has changed to whatever you committed above.

And then you’re done! You’ve created an end-to-end GitOps continuous delivery pipeline from code to cluster that requires no intervention other than a code change!

Cleanup

Don’t forget to terraform destroy your cluster, to avoid incurring a large bill with your cloud provider!

Lessons Learned

Even though this is as simple an example I could make, you can see that it involves quite a bit of configuration and setup. If anything went wrong in the pipeline, you’d need to understand quite a lot to be able to debug it and fix it.

In addition, there are numerous design decisions you need to make to get your GitOps workflow working for you. Some of these are covered in my previous GitOps Decisions post.

Anyone who’s coded anything will know that there is this tax on the benefits of automation. It should not be underestimated by any team looking to take up this deployment methodology.

On the other hand, once the mental model of the workflow is internalised by a team, significant savings and improvements to delivery are seen.

---

If you like this, you might like one of my books:
Learn Bash the Hard Way
Learn Git the Hard Way
Learn Terraform the Hard Way

---

If you enjoyed this, then please consider buying me a coffee to encourage me to do more.

---