Infrastructure as Code with Terraform and GitHub Actions: A Kubernetes Case Study

As I’ve been working with Terraform more and more these days, I felt that it would be a good idea to move away from some of my other hacked together solutions in favor of Terraform. My next logical step was to focus on kubernetes management with terraform and github actions. This idea builds upon my previous Using Github to Manage Kubernetes article. As I started down this path, I realized that I needed a way to manage my state file. This is why I thought it was important to tackle the configuration I posted in Managing Your Terraform State File.

GitHub Secrets Setup

I can now focus on making Terraform manage my Kubernetes via GitHub Actions because my state file is managed in AWS. In order to do this, I needed to add some secrets into my GitHub repository. The GitHub Secrets page on the GitHub site documents how to create these secrets, so I will not cover it here. The table below covers the secrets that I created:

Creating the Terraform GitHub Action

Next, I’m setting up my GitHub Action for Terraform. We start with the basic configuration

name: Terraform plan, validate, and apply

env:
  tf_actions_working_dir: 'terraform'
  TF_VAR_cluster_endpoint: ${{ secrets.TF_VAR_CLUSTER_ENDPOINT }}
  TF_VAR_cluster_user_token: ${{ secrets.TF_VAR_CLUSTER_USER_TOKEN }}
  TF_VAR_cluster_ca_cert: ${{ secrets.TF_VAR_CLUSTER_CA_CERT }}
  AWS_DEFAULT_REGION: "us-east-2"
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

on:
    # Triggers the workflow on push request on the main branch for changes in the specified paths.
    push:
      branches:
        - main
      paths:
        - '.github/workflows/terraform.yml'
        - 'terraform/*'

The name is not important and you can call it whatever you like. Next, we have the env section where I’m passing all of the secrets that I created as Environment Variables to the GitHub Action.

My CI/CD is not an overly complex flow of branches and PRs at this time since I’m an army of one so I’ll only worry about performing this GitHub Action on pushes to the main branch. I am limiting this Action by only executing it whenever a change is made with the terraform directory of the repo or this Actions file.

In a more sophisticated environment, I would suggest breaking out this flow so that we can perform plan/validate on PRs and apply only when the PR is merged.

Now, we add the jobs to the Action

name: Terraform plan, validate, and apply

env:
  tf_actions_working_dir: 'terraform'
  TF_VAR_cluster_endpoint: ${{ secrets.TF_VAR_CLUSTER_ENDPOINT }}
  TF_VAR_cluster_user_token: ${{ secrets.TF_VAR_CLUSTER_USER_TOKEN }}
  TF_VAR_cluster_ca_cert: ${{ secrets.TF_VAR_CLUSTER_CA_CERT }}
  AWS_DEFAULT_REGION: "us-east-2"
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

on:
    # Triggers the workflow on push request on the main branch for changes in the specified paths.
    push:
      branches:
        - main
      paths:
        - '.github/workflows/terraform.yml'
        - 'terraform/*'

# A workflow run is made up of one or more jobs that can run sequentially or in parallel.
jobs:
  # This workflow contains a single job called "build".
  build:
    # The type of runner that the job will run on.
    runs-on: ubuntu-latest
    
    defaults:
      run:
        working-directory: ${{ env.tf_actions_working_dir }}
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it.
    - name: Checkout main
      uses: actions/checkout@main

    - uses: hashicorp/setup-terraform@v2
    
    - name: Terraform Init
      id: init
      run: terraform init

    - name: Terraform fmt
      id: fmt
      run: terraform fmt -check
      continue-on-error: true

    - name: Terraform Validate
      id: validate
      run: terraform validate -no-color
    
    - name: Terraform Plan
      id: plan
      run: terraform plan -no-color
      continue-on-error: true
    
    - name: Terraform Apply
      id: apply
      run: terraform apply -no-color -auto-approve

The environment is running on ubuntu and I’m forcing the working directory to be my Terraform directory in the GitHub repository. Aside from the actions/checkout, I’m making use of Hashicorp’s Terraform Github Action, setup-terraform. I’ll explain each of the steps below for this Action.

Terraform GitHub Action Steps

The uses statement tells this GitHub Action to use Hashicorp’s Terraform GitHub Action. From there, our first step is the terraform init which will setup the Terraform environment in the ubuntu runner.

Next up, we’re doing some formatting and validation with the fmt and validate Terraform commands. The fmt command is using the <a href="https://developer.hashicorp.com/terraform/cli/commands/fmt#check">-check</a> switch which will exit if there are changes that need to be made to properly format the modules. The <a href="https://developer.hashicorp.com/terraform/cli/commands/validate">validate</a> command will validate the Terraform files locally. There are no remote calls to any services during this validation.

Next, we do a plan to connect to the remote endpoints and determine what changes need to be made and print those out.

Finally, we apply to make the changes defined in the Terraform modules. In this apply step, I’m adding -auto-approve to the terraform apply command so that the changes are automatically applied without prompting for me to approve them. This is quite important for automation.

Creating a Terraform to Manage Kubernetes

With everything in place on the GitHub Actions side, it is time to create my Terraform to be used by GitHub Actions to manage my Kubernetes. I already have a backend.tf setup in this repository that has been configured to manage my state file in AWS like I described in Managing Your Terraform State File.

Defining Terraform Variables

I want to first define some input variables to my Terraform that will make use of the environment variables created above. I created a variables.tf that looks like

variable "cluster_endpoint" {
  type = string
}

variable "cluster_ca_cert" {
  type = string
}

variable "cluster_user_token" {
  type = string
}

Notice the names of these variables is the same as the ones listed in the env section of my GitHub Action. The only difference is that they are prepended with TF_VAR_.

env:
...
  TF_VAR_cluster_endpoint: ${{ secrets.TF_VAR_CLUSTER_ENDPOINT }}

This env statement is telling the GitHub Action to set an environment variable called TF_VAR_cluster_endpoint that is equal to the value of the TF_VAR_CLUSTER_ENDPOINT secret. When we run our terraform commands, Terraform will then populate the value of TF_VAR_cluster_endpoint into the cluster_endpoint input variable in our Terraform module (var.cluster_endpoint).

Configuring Our Terraform Provider

The next file I’ve created is the provider.tf file

provider "kubernetes" {
  host                   = var.cluster_endpoint
  cluster_ca_certificate = base64decode(var.cluster_ca_cert)
  token                  = var.cluster_user_token
}

I’m telling Terraform to use the Kubernetes provider. This provider requires some configuration options so that it knows which Kubernetes cluster to manage. I’m supplying the required parameters via my input variables.

Creating Kubernetes Resources

The next step is to define various the Kubernetes resources that we’d like to have created. I’m starting with a very simple namepsace definition that I’m placing in my namespaces.tf file

resource "kubernetes_namespace" "example" {
  metadata {
    name = "my-first-namespace"
  }
}

This is using the kubernetes_namespace resource to define a new Kubernetes Namespace in my cluster.

Putting The Kubernetes Management with Terraform and GitHub Actions into Action

We’ve done all of the setup so now let’s see what happens. I do a commit and push and check out the Actions (Yes, I know there’s a few more steps in there than what I defined above but I’m still tinkering).

% kubectl get ns
NAME                 STATUS   AGE
ctesting             Active   643d
cyral-0b9yxo         Active   277d
default              Active   3y355d
kube-node-lease      Active   3y355d
kube-public          Active   3y355d
kube-system          Active   3y355d
my-first-namespace   Active   2m20s

Holy crap! My cluster has been around for nearly 4yrs! Aside from that, I can also see my namespace that I created, my-first-namespace.

The Wrap Up

This is just a very simple example of managing your Kubernetes through Terraform. I can see me migrating some of my other stuff like what I created in Using Github to Manage Kubernetes to this flow instead.

As I mentioned earlier in the article, I have a very basic flow and nothing too complex. In a more production environment, I would suggest breaking these into a few more steps rather than a single commit to main. You could introduce a precommit like I described in Adding pre-commit Hooks to Python Repo. This precommit could perform some validation like JSON formatting and even the fmt and validate. You could even move the fmt and validate into the commit phase.

I would suggest adding terraform plan to PRs and then leave only apply for merges into main.

GitHub Secrets Setup#

Creating the Terraform GitHub Action#

Terraform GitHub Action Steps#

Creating a Terraform to Manage Kubernetes#

Defining Terraform Variables#

Configuring Our Terraform Provider#

Creating Kubernetes Resources#

Putting The Kubernetes Management with Terraform and GitHub Actions into Action#

The Wrap Up#