Building a Kubernetes Container That Synchs with Private Git Repo

My previous post explained how to create a private git repo. On its own, that post is roughly useless unless you planned to maintained some private copy of your project so nobody can see it. In this post, we’re going to put that private repo to use in a Kubernetes environment. A basic assumption is that you already have a Kubernetes environment setup.

Adding Another SSH Key to the Repo

The first step would be to add another SSH Key to our repo. The purpose of this key is to be used to configure access from the container to the repo. We’ll load the SSH key into Kubernetes as a secret. We can’t set a password on this key or we might get prompted for the password during container build and that’s not useful. Also, since the key will not have a password, we won’t give it Read / Write access to our repo.

Generate the SSH Key

As before, we’re going to run the ssh-keygen command but we’ll specify the file where to save the key and just simply hit enter at the password prompt so that it’s not password protected.

imacs-imac:~ scott$ ssh-keygen -t rsa
 Generating public/private rsa key pair.
 Enter file in which to save the key (/Users/scott/.ssh/id_rsa): /Users/scott/.ssh/GH_RO_key_rsa
 Enter passphrase (empty for no passphrase): 
 Enter same passphrase again: 
 Your identification has been saved in /Users/scott/.ssh/GH_RO_key_rsa.
 Your public key has been saved in /Users/scott/.ssh/GH_RO_key_rsa.pub.
 The key fingerprint is:
 SHA256:0v0koHVNHdJbt4j2PaNorHa25dXgNl0sQjJB8R3ClPA [email protected]
 The key's randomart image is:
 +---[RSA 2048]----+
 |         .===+o. |
 |           *o+o.o|
 |        o + E ooo|
 |       + + * ..o |
 |      o S + + + o|
 |       .   + + Bo|
 |          . o.=.=|
 |         . *oo.. |
 |        ..=...   |
 +----[SHA256]-----+
 imacs-imac:~ scott$

Upload the Key to our Git Repo

With our new SSH Key created, we’ll want to once again take the contents of the .pub file aka GH_RO_key_rsa.pub if you’re following along and paste that into our repo’s Deploy Keys like below:

Now that we have our new Read Only key added to the repo, it’s time to setup Kubernetes. This is going to be a simple configuration so that we can display static HTML pages on our Kubernetes cluster.

Add SSH Key to Kubernetes

In order to have Kubernetes be able to use the SSH key, we need to add it as a secret that we’ll reference in our pod deployment. The first step is to create a known hosts file to be used along with the key so we don’t have to worry about acknowledging any new key messages.

~# ssh-keyscan github.com > /tmp/known_hosts
 # github.com:22 SSH-2.0-babeld-778045a0
 # github.com:22 SSH-2.0-babeld-778045a0
 # github.com:22 SSH-2.0-babeld-778045a0
 ~#

This copies the ssh key from github into the /tmp/known_hosts file. Next, we need to get the contents of our private key file. When we pasted the key into GitHub, we were working with the public key file..aka the .pub file…Since Kubernetes will need to authenticate using this key, it’ll need the private key file…aka the GH_RO_key_rsa file. We’ll use the kubectl command to add the key into Kubernetes:

~# kubectl create secret generic github-creds --from-file=ssh=.ssh/GH_RO_key_rsa --from-file=known_hosts=/tmp/known_hosts
 secret/github-creds created
 ~#

Creating the Web Server Deployment

Now we’re going to create a YAML file to configure and setup everything. The start of that YAML file will be to configure Kubernetes to open a port that directs traffic to port 80 of our resulting pod. From there, we’ll need to setup a pod that runs two separate containers. One container will be our git-synch application and the other will be nginx. We could get into some “complex” discussions and added costs of running a PVC or some other Kubernetes shared storage but we’re only dealing with a small web site that is synched with github so we’re gonna simply leverage local storage on each node by defining two volumes:

      volumes:
       - name: dir
         emptyDir: {}
       - name: git-secret
         secret:
           secretName: github-creds
           defaultMode: 288

This creates two volumes dir and git-secret. The dir is simply an empty directory volume that we’ll be filling with our files that we synch from Github. The git-secret is the SSH Key we added above. This needs to be made available to our git-synch container.

In the nginx container, we’re going to mount the dir volume as /usr/share/nginx. The default nginx image looks for web content, aka document root, in /usr/share/nginx/html. Therefore, we’re going mount the repo as /usr/share/nginx. We mount the dir volume to /git as this is where we’re going to write our synched data.

You can see all of these configurations in the git-synch container configuration such as the target location for our synched files as well as the secret to use.

      containers:
       - env:
         - name: GIT_SYNC_REPO
           value: [email protected]:<some user>/mysamplerepo.git
         - name: GIT_SYNC_BRANCH
           value: master
         - name: GIT_SYNC_SSH
           value: "true"
         - name: GIT_SYNC_PERMISSIONS
           value: "0777"
         - name: GIT_SYNC_DEST
           value: www
         - name: GIT_SYNC_ROOT
           value: /git
         name: git-sync
         image: k8s.gcr.io/git-sync:v3.1.1
         securityContext:
           runAsUser: 65533 # git-sync user
         volumeMounts:
         - name: git-secret
           mountPath: /etc/git-secret
         - name: dir
           mountPath: /git

You’ll want to make sure you change the GIT_SYNC_REPO to match the value of your clone/download link in Github. The GIT_SYNC_DEST should match the name of your repo.

Here is the full config for reference:

apiVersion: v1
 kind: Service
 metadata:
   name: webserver
   labels:
     tier: backend
 spec:
   selector:
     app: webserver
     tier: backend
   ports:  
   - name: http
     port: 80
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: webserver
   labels:
     tier: backend
 spec:
   replicas: 1
   selector:
     matchLabels:
       app: webserver
       tier: backend
   template:
     metadata:
       labels:
         app: webserver
         tier: backend
     spec:
       securityContext:
         fsGroup: 65533 # to make SSH key readable
       volumes:
       - name: dir
         emptyDir: {}
       - name: git-secret
         secret:
           secretName: github-creds
           defaultMode: 288
       containers:
       - env:
         - name: GIT_SYNC_REPO
           value: [email protected]:<some user>/mysamplerepo.git
         - name: GIT_SYNC_BRANCH
           value: master
         - name: GIT_SYNC_SSH
           value: "true"
         - name: GIT_SYNC_PERMISSIONS
           value: "0777"
         - name: GIT_SYNC_DEST
           value: www
         - name: GIT_SYNC_ROOT
           value: /git
         name: git-sync
         image: k8s.gcr.io/git-sync:v3.1.1
         securityContext:
           runAsUser: 65533 # git-sync user
         volumeMounts:
         - name: git-secret
           mountPath: /etc/git-secret
         - name: dir
           mountPath: /git
       - name: webserver
         image: nginx:latest
         ports:
         - containerPort: 80
         volumeMounts:
         - name: dir
           mountPath: /usr/share/nginx

With out configuration file all ready to go, we’ll use kubectl to apply the file:

~# kubectl apply -f webserver.yaml 
 service/webserver created
 deployment.apps/webserver created
 ~#

After some time, we should be able to check the status and see the pod is online and the service is setup:

~# kubectl get pod
 NAME                         READY   STATUS    RESTARTS   AGE
 webserver-686854f667-cwq5f   2/2     Running   5          3m46s
 ~# kubectl get svc
 NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
 kubernetes   ClusterIP   10.152.183.1     <none>        443/TCP   149m
 webserver    ClusterIP   10.152.183.195   <none>        80/TCP    5m28s
 ~#

Testing the Deployment

With everything deployed, we should have a web server up and running that is serving our git repo from the previous post. Without getting into deploying an ingress server and such, let’s take a short cut to test out our deployment. We can do this by connecting to the web server and doing a curl. First, we connect to the web server container:

# kubectl exec -it webserver-686854f667-cwq5f -c webserver /bin/bash

The above command will connect you to a shell in the container. By default, the nginx image does not have curl installed so we’ll need to install this to test further. Install curl using the below commands:

root@webserver-686854f667-cwq5f:/# apt update;apt -y install curl

With curl installed, let’s connect to the local web server:

root@webserver-686854f667-cwq5f:/# curl localhost
 <html>
 <head><title>403 Forbidden</title></head>
 <body>
 <center><h1>403 Forbidden</h1></center>
 <hr><center>nginx/1.17.6</center>
 </body>
 </html>

That does not seem right…I broke something…didn’t I? Oh wait, I know let’s try…

root@webserver-686854f667-cwq5f:/# curl localhost/html/
 <html>
 <body>
 hello world!
 </body>
 </html>

That works better. Looks like we need to fix something here but first let’s see if making a change to the repo works. Let’s cheat and use the github file editor and make a change to the index.html file like the below:

root@webserver-686854f667-cwq5f:/# curl localhost/html/
 <html>
 <body>
 hello world! Test #2
 </body>
 </html>

Boom! Just like that it’s working. Kinda…

Fixing Our Deployment

In case the problem isn’t quite obvious, we are attempting to mount the git repo in a location that nginx isn’t quite looking for. It’s a bad idea to mount the entire git repo as the document root since it could allow people to look at your .git directory and possibly other files that you didn’t consider. In order to fix our deployment and secure just a little further we’re going to first adjust the nginx configuration with a Kubernetes configmap:

apiVersion: v1
 kind: ConfigMap
 metadata:
   name: webserver-config
   labels:
     tier: backend
 data:
   config :
     server {
         listen       80;
         server_name  localhost;
     
         location / {
             root   /usr/share/nginx/www/html;
             index  index.html index.htm;
         }
     
         error_page   500 502 503 504  /50x.html;
         location = /50x.html {
             root   /usr/share/nginx/www/html;
         }
    
     }

This configmap supplies nginx with a new configuration for the default site that tells nginx that the document root is now located in /usr/share/nginx/www/html. We also made some changes to the original webserver.yaml to add this new configuration as well as changing the mount point for git and nginx. The full configuration is here.

apiVersion: v1
 kind: ConfigMap
 metadata:
   name: webserver-config
   labels:
     tier: backend
 data:
   config :
     server {
         listen       80;
         server_name  localhost;
         location / {
             root   /usr/share/nginx/www/html;
             index  index.html index.htm;
         }
         error_page   500 502 503 504  /50x.html;
         location = /50x.html {
             root   /usr/share/nginx/www/html;
         }
     }
 ---
 apiVersion: v1
 kind: Service
 metadata:
   name: webserver
   labels:
     tier: backend
 spec:
   selector:
     app: webserver
     tier: backend
   ports:  
   - name: http
     port: 80
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: webserver
   labels:
     tier: backend
 spec:
   replicas: 1
   selector:
     matchLabels:
       app: webserver
       tier: backend
   template:
     metadata:
       labels:
         app: webserver
         tier: backend
     spec:
       securityContext:
         fsGroup: 65533 # to make SSH key readable
       volumes:
       - name: dir
         emptyDir: {}
       - name: git-secret
         secret:
           secretName: github-creds
           defaultMode: 288
       - name: config
         configMap:
           name: webserver-config
           items:
           - key: config
             path: default.conf
       containers:
       - env:
         - name: GIT_SYNC_REPO
           value: [email protected]:<some user>/mysamplerepo.git
         - name: GIT_SYNC_BRANCH
           value: master
         - name: GIT_SYNC_SSH
           value: "true"
         - name: GIT_SYNC_PERMISSIONS
           value: "0777"
         - name: GIT_SYNC_DEST
           value: www
         - name: GIT_SYNC_ROOT
           value: /git
         name: git-sync
         image: k8s.gcr.io/git-sync:v3.1.1
         securityContext:
           runAsUser: 65533 # git-sync user
         volumeMounts:
         - name: git-secret
           mountPath: /etc/git-secret
         - name: dir
           mountPath: /git
       - name: webserver
         image: nginx:latest
         ports:
         - containerPort: 80
         volumeMounts:
         - name: dir
           mountPath: /usr/share/nginx
         - name: config
           mountPath: /etc/nginx/conf.d

Let’s apply this updated configuration using kubectl:

root@do-nyc04:/tmp# kubectl apply -f webserver.yaml 
 configmap/webserver created
 service/webserver unchanged
 deployment.apps/webserver configured

Let’s now reconnect and test our configuration:

root@do-nyc04:/tmp# kubectl get pod -o wide
 NAME                         READY   STATUS    RESTARTS   AGE    IP             NODE              NOMINATED NODE   READINESS GATES
 webserver-8fb84dc86-5chm5    2/2     Running   0          17s    10.244.1.53    pool-sfo01-ssy1   <none>           <none>
 root@do-nyc04:/tmp# kubectl exec -it webserver-8fb84dc86-5chm5 -c webserver /bin/bash
 root@webserver-8fb84dc86-5chm5:/# apt update;apt -y install curl
 Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
 Get:2 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]             
 Get:3 http://security-cdn.debian.org/debian-security buster/updates InRelease [65.4 kB]
 Get:4 http://deb.debian.org/debian buster/main amd64 Packages [7908 kB]
 Get:5 http://deb.debian.org/debian buster-updates/main amd64 Packages [5792 B]
 Get:6 http://security-cdn.debian.org/debian-security buster/updates/main amd64 Packages [167 kB]
 Fetched 8317 kB in 2s (3534 kB/s)                         
 Reading package lists... Done
 Building dependency tree       
 Reading state information... Done
 All packages are up to date.
 Reading package lists... Done
 Building dependency tree       
 Reading state information... Done
 The following additional packages will be installed:
 ...
 128 added, 0 removed; done.
 Setting up libgssapi-krb5-2:amd64 (1.17-3) ...
 Setting up libcurl4:amd64 (7.64.0-4) ...
 Setting up curl (7.64.0-4) ...
 Processing triggers for libc-bin (2.28-10) ...
 Processing triggers for ca-certificates (20190110) ...
 Updating certificates in /etc/ssl/certs...
 0 added, 0 removed; done.
 Running hooks in /etc/ca-certificates/update.d...
 done.
  
 root@webserver-8fb84dc86-5chm5:/# curl localhost
 <html>
 <body>
 hello world! Test #2
 </body>
  
 </html>

Great news! It looks like it’s fixed. Just to make sure things are working still, let’s make another change and see if it publishes.

root@webserver-8fb84dc86-5chm5:/# curl localhost
 <html>
 <body>
 hello world! Everything must be cleaned up at this point
 </body>
 </html>

W00t! Looks like everything is working and as we expect. Although, this configuration is mostly useless unless you are actually within the Kubernetes cluster. For the next article, I’ll provide some options and a hack for exposing this web server to the world.

Adding Another SSH Key to the Repo#

Generate the SSH Key#

Upload the Key to our Git Repo#

Add SSH Key to Kubernetes#

Creating the Web Server Deployment#

Testing the Deployment#

Fixing Our Deployment#