As I’ve been working with Terraform more and more these days, I felt that it would be a good idea to move away from some of my other hacked together solutions in favor of Terraform. My next logical step was to focus on kubernetes management with terraform and github actions. This idea builds upon my previous Using Github to Manage Kubernetes article. As I started down this path, I realized that I needed a way to manage my state file. This is why I thought it was important to tackle the configuration I posted in Managing Your Terraform State File.
GitHub Secrets Setup
I can now focus on making Terraform manage my Kubernetes via GitHub Actions because my state file is managed in AWS. In order to do this, I needed to add some secrets into my GitHub repository. The GitHub Secrets page on the GitHub site documents how to create these secrets, so I will not cover it here. The table below covers the secrets that I created:
| Secret Name | Secret Description |
|---|---|
| AWS_ACCESS_KEY_ID | This is the AWS Access Key ID created for the IAM User I used when setting up my Terraform state information in AWS. |
| AWS_SECRET_ACCESS_KEY | This is the AWS Secret created for the IAM User I used when setting up my Terraform state information in AWS. |
| TF_VAR_CLUSTER_ENDPOINT | This is my Kubernetes cluster’s management endpoint |
| TF_VAR_CLUSTER_USER_TOKEN | This is the user token that I’ll be using to manage my Kubernetes |
| TF_VAR_CLUSTER_CA_CERT | This is the ca cert for the user that I’ll be using to manage my Kubernetes |
If you have dealt with AWS, you should be familiar with the AWS information. If not, here’s a link to the AWS Users documentation for additional context. The variables that begin with TF_VAR_ are actually used as Terraform Input Variables. You can provide inputs to Terraform in a number of ways. One of those ways is via Environment Variables. By creating the above secrets, I will be supplying them to my GitHub Action for Terraform so that it will pass them along to my Terraform modules.
Creating the Terraform GitHub Action
Next, I’m setting up my GitHub Action for Terraform. We start with the basic configuration
name: Terraform plan, validate, and apply
env:
tf_actions_working_dir: 'terraform'
TF_VAR_cluster_endpoint: ${{ secrets.TF_VAR_CLUSTER_ENDPOINT }}
TF_VAR_cluster_user_token: ${{ secrets.TF_VAR_CLUSTER_USER_TOKEN }}
TF_VAR_cluster_ca_cert: ${{ secrets.TF_VAR_CLUSTER_CA_CERT }}
AWS_DEFAULT_REGION: "us-east-2"
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
on:
# Triggers the workflow on push request on the main branch for changes in the specified paths.
push:
branches:
- main
paths:
- '.github/workflows/terraform.yml'
- 'terraform/*'
The name is not important and you can call it whatever you like. Next, we have the env section where I’m passing all of the secrets that I created as Environment Variables to the GitHub Action.
My CI/CD is not an overly complex flow of branches and PRs at this time since I’m an army of one so I’ll only worry about performing this GitHub Action on pushes to the main branch. I am limiting this Action by only executing it whenever a change is made with the terraform directory of the repo or this Actions file.
In a more sophisticated environment, I would suggest breaking out this flow so that we can perform plan/validate on PRs and apply only when the PR is merged.
Now, we add the jobs to the Action
name: Terraform plan, validate, and apply
env:
tf_actions_working_dir: 'terraform'
TF_VAR_cluster_endpoint: ${{ secrets.TF_VAR_CLUSTER_ENDPOINT }}
TF_VAR_cluster_user_token: ${{ secrets.TF_VAR_CLUSTER_USER_TOKEN }}
TF_VAR_cluster_ca_cert: ${{ secrets.TF_VAR_CLUSTER_CA_CERT }}
AWS_DEFAULT_REGION: "us-east-2"
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
on:
# Triggers the workflow on push request on the main branch for changes in the specified paths.
push:
branches:
- main
paths:
- '.github/workflows/terraform.yml'
- 'terraform/*'
# A workflow run is made up of one or more jobs that can run sequentially or in parallel.
jobs:
# This workflow contains a single job called "build".
build:
# The type of runner that the job will run on.
runs-on: ubuntu-latest
defaults:
run:
working-directory: ${{ env.tf_actions_working_dir }}
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it.
- name: Checkout main
uses: actions/checkout@main
- uses: hashicorp/setup-terraform@v2
- name: Terraform Init
id: init
run: terraform init
- name: Terraform fmt
id: fmt
run: terraform fmt -check
continue-on-error: true
- name: Terraform Validate
id: validate
run: terraform validate -no-color
- name: Terraform Plan
id: plan
run: terraform plan -no-color
continue-on-error: true
- name: Terraform Apply
id: apply
run: terraform apply -no-color -auto-approve
The environment is running on ubuntu and I’m forcing the working directory to be my Terraform directory in the GitHub repository. Aside from the actions/checkout, I’m making use of Hashicorp’s Terraform Github Action, setup-terraform. I’ll explain each of the steps below for this Action.
Terraform GitHub Action Steps
The uses statement tells this GitHub Action to use Hashicorp’s Terraform GitHub Action. From there, our first step is the terraform init which will setup the Terraform environment in the ubuntu runner.
Next up, we’re doing some formatting and validation with the fmt and validate Terraform commands. The fmt command is using the -check switch which will exit if there are changes that need to be made to properly format the modules. The validate command will validate the Terraform files locally. There are no remote calls to any services during this validation.
Next, we do a plan to connect to the remote endpoints and determine what changes need to be made and print those out.
Finally, we apply to make the changes defined in the Terraform modules. In this apply step, I’m adding -auto-approve to the terraform apply command so that the changes are automatically applied without prompting for me to approve them. This is quite important for automation.
Creating a Terraform to Manage Kubernetes
With everything in place on the GitHub Actions side, it is time to create my Terraform to be used by GitHub Actions to manage my Kubernetes. I already have a backend.tf setup in this repository that has been configured to manage my state file in AWS like I described in Managing Your Terraform State File.
Defining Terraform Variables
I want to first define some input variables to my Terraform that will make use of the environment variables created above. I created a variables.tf that looks like
variable "cluster_endpoint" {
type = string
}
variable "cluster_ca_cert" {
type = string
}
variable "cluster_user_token" {
type = string
}
Notice the names of these variables is the same as the ones listed in the env section of my GitHub Action. The only difference is that they are prepended with TF_VAR_.
env:
...
TF_VAR_cluster_endpoint: ${{ secrets.TF_VAR_CLUSTER_ENDPOINT }}
This env statement is telling the GitHub Action to set an environment variable called TF_VAR_cluster_endpoint that is equal to the value of the TF_VAR_CLUSTER_ENDPOINT secret. When we run our terraform commands, Terraform will then populate the value of TF_VAR_cluster_endpoint into the cluster_endpoint input variable in our Terraform module (var.cluster_endpoint).
Configuring Our Terraform Provider
The next file I’ve created is the provider.tf file
provider "kubernetes" {
host = var.cluster_endpoint
cluster_ca_certificate = base64decode(var.cluster_ca_cert)
token = var.cluster_user_token
}
I’m telling Terraform to use the Kubernetes provider. This provider requires some configuration options so that it knows which Kubernetes cluster to manage. I’m supplying the required parameters via my input variables.
Creating Kubernetes Resources
The next step is to define various the Kubernetes resources that we’d like to have created. I’m starting with a very simple namepsace definition that I’m placing in my namespaces.tf file
resource "kubernetes_namespace" "example" {
metadata {
name = "my-first-namespace"
}
}
This is using the kubernetes_namespace resource to define a new Kubernetes Namespace in my cluster.
Putting The Kubernetes Management with Terraform and GitHub Actions into Action
We’ve done all of the setup so now let’s see what happens. I do a commit and push and check out the Actions (Yes, I know there’s a few more steps in there than what I defined above but I’m still tinkering).
It looks like the Action completed successfully so let’s check our Kubernetes cluster for my namespace.
% kubectl get ns
NAME STATUS AGE
ctesting Active 643d
cyral-0b9yxo Active 277d
default Active 3y355d
kube-node-lease Active 3y355d
kube-public Active 3y355d
kube-system Active 3y355d
my-first-namespace Active 2m20s
Holy crap! My cluster has been around for nearly 4yrs! Aside from that, I can also see my namespace that I created, my-first-namespace.
The Wrap Up
This is just a very simple example of managing your Kubernetes through Terraform. I can see me migrating some of my other stuff like what I created in Using Github to Manage Kubernetes to this flow instead.
As I mentioned earlier in the article, I have a very basic flow and nothing too complex. In a more production environment, I would suggest breaking these into a few more steps rather than a single commit to main. You could introduce a precommit like I described in Adding pre-commit Hooks to Python Repo. This precommit could perform some validation like JSON formatting and even the fmt and validate. You could even move the fmt and validate into the commit phase.
I would suggest adding terraform plan to PRs and then leave only apply for merges into main.