From Reactive to Proactive: Transforming Security with Tetragon

by
PinLinkedIn

I previously blogged about Starboard and How to Install and Use Starboard to Protect Your Kubernetes Cluster. These articles were focused more on vulnerability and configuration management. Now, I wanted to focus my attention on runtime security observability using Tetragon.

Getting Started With Tetragon

The first step is to install it. The Tetragon website recommends using Helm 3 to deploy it so that’s what we’ll do. I’m deploying with just the default values for now 

 % helm repo add cilium https://helm.cilium.io
"cilium" has been added to your repositories

% helm repo update

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "aquasecurity" chart repository
...Successfully got an update from the "cilium" chart repository
Update Complete. โŽˆHappy Helming!โŽˆ

% helm install tetragon cilium/tetragon -n kube-system

NAME: tetragon
LAST DEPLOYED: Sun Dec 17 13:48:00 2023
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

%

After that, we’ll check on the install status: 

 % kubectl rollout status -n kube-system ds/tetragon -w
daemon set "tetragon" successfully rolled out

% kubectl -n kube-system get pod
NAME                                 READY   STATUS    RESTARTS      AGE
cilium-6zrtw                         1/1     Running   0             22d
cilium-7fzqg                         1/1     Running   0             22d
cilium-operator-684fc7cc47-nqgs5     1/1     Running   0             22d
cilium-v4lt7                         1/1     Running   0             22d
...
tetragon-2twpw                       2/2     Running   0             87s
tetragon-b2d66                       2/2     Running   0             87s
tetragon-operator-5f5489bfd9-xkrcn   1/1     Running   0             87s
tetragon-wsgmg                       2/2     Running   0             87s

It looks like we’re up and running so now what should I do?

Checking Out the Execution Monitoring

I’ve been using my prod-mysql-0 pod in the ctesting namespace as my guinea pig for everything. Let’s try getting the execution events for this pod: 

% kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --pods prod-mysql-0
๐Ÿš€ process ctesting/prod-mysql-0 /bin/bash -ec "password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
" 
๐Ÿš€ process ctesting/prod-mysql-0 /opt/bitnami/mysql/bin/mysqladmin status -uroot -<super_secret_password> 
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /opt/bitnami/mysql/bin/mysqladmin status -uroot -<super_secret_password> 0 
๐Ÿš€ process ctesting/prod-mysql-0 /bin/bash -ec "password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
" 
๐Ÿš€ process ctesting/prod-mysql-0 /opt/bitnami/mysql/bin/mysqladmin status -uroot -<super_secret_password> 
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /opt/bitnami/mysql/bin/mysqladmin status -uroot -<super_secret_password> 0

That’s awesome! What does it mean? The short version is that I’m seeing the commands being executed by the container itself. This does not count commands executed when clients connect to the database. By the looks of this, I’m seeing execution of the status check for the container. It’s trying to get the root password from the environment variable MYSQL_ROOT_PASSWORD and then checks for a password file to get it from. From there, it runs the status command. This all seems normal.

Let’s try for something a little abnormal. I kept the monitoring going and then connected with exec to the container like this: 

% kubectl exec -it -n ctesting  prod-mysql-0 -- /bin/bash 
Defaulted container "mysql" out of: mysql, git-sync (init)
I have no name!@prod-mysql-0:/$ curl www.google.com
bash: curl: command not found
I have no name!@prod-mysql-0:/$ wget www.google.com
bash: wget: command not found
I have no name!@prod-mysql-0:/$ ls -al
total 88
drwxr-xr-x   1 root root 4096 Dec  9 12:54 .
drwxr-xr-x   1 root root 4096 Dec  9 12:54 ..
drwxrwxr-x   2 root root 4096 Oct 29 16:30 .mysqlsh
drwxr-xr-x   2 root root 4096 Oct 29 16:28 bin
drwxr-xr-x   3 root root 4096 Oct 29 16:30 bitnami
drwxr-xr-x   2 root root 4096 Apr 18  2023 boot
drwxr-xr-x   5 root root  360 Dec  9 12:54 dev
drwxrwsrwx   3 root 1001 4096 Dec  9 12:54 docker-entrypoint-initdb.d
drwxr-xr-x   1 root root 4096 Dec  9 12:54 etc
drwxr-xr-x   2 root root 4096 Apr 18  2023 home
drwxr-xr-x   7 root root 4096 Oct 29 16:28 lib
drwxr-xr-x   2 root root 4096 Apr 18  2023 lib64
drwxr-xr-x   2 root root 4096 Apr 18  2023 media
drwxr-xr-x   2 root root 4096 Apr 18  2023 mnt
drwxrwxr-x   1 root root 4096 Dec  9 12:54 opt
dr-xr-xr-x 239 root root    0 Dec  9 12:54 proc
drwx------   2 root root 4096 Apr 18  2023 root
drwxr-xr-x   1 root root 4096 Dec  9 12:54 run
drwxr-xr-x   2 root root 4096 Oct 29 16:28 sbin
drwxr-xr-x   2 root root 4096 Apr 18  2023 srv
dr-xr-xr-x  13 root root    0 Dec  9 12:54 sys
drwxrwxrwt   1 root root 4096 Dec  9 12:54 tmp
drwxrwxr-x  11 root root 4096 Oct 29 16:29 usr
drwxr-xr-x  11 root root 4096 Oct 29 16:29 var
I have no name!@prod-mysql-0:/$ 
exit

Now let’s look at the execution events 

๐Ÿš€ process ctesting/prod-mysql-0 /opt/bitnami/mysql/bin/mysqladmin status -uroot -<super_secret_password> 
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /opt/bitnami/mysql/bin/mysqladmin status -uroot -<super_secret_password> 0 
๐Ÿš€ process ctesting/prod-mysql-0 /bin/bash -ec "password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
" 
๐Ÿš€ process ctesting/prod-mysql-0 /opt/bitnami/mysql/bin/mysqladmin status -uroot -<super_secret_password> 
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /opt/bitnami/mysql/bin/mysqladmin status -uroot -<super_secret_password> 0 
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /bin/bash  127                  
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /bin/bash  127                  
๐Ÿš€ process ctesting/prod-mysql-0 /bin/ls -al                              
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /bin/ls -al 0                   
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /bin/bash  0                    
๐Ÿš€ process ctesting/prod-mysql-0 /bin/bash -ec "password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
"

Now THAT seems like a lot more fun! We can see that there are some additional commands being executed that we would not expect from the container. This is useful.

Checking Out the File Access Monitoring

Continuing the exploration, I also wanted to check out the file access monitoring. This is not enabled by default and requires you to provide a YAML file that lists out the files to be monitored. For this initial runtime security observability exercise, I’ll use the quickstart YAML file provided by Tetragon: 

% kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml
tracingpolicy.cilium.io/file-monitoring-filtered created

With this in place, I’ll try accessing one of the monitored files /etc/shadow: 

% kubectl exec -it -n ctesting  prod-mysql-0 -- /bin/bash
Defaulted container "mysql" out of: mysql, git-sync (init)
I have no name!@prod-mysql-0:/$ cat /etc/shadow
cat: /etc/shadow: Permission denied
I have no name!@prod-mysql-0:/$ 
exit
command terminated with exit code 1

While I was running the above commands, I also ran the gettevents command: 

๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /bin/bash  1                    
๐Ÿš€ process ctesting/prod-mysql-0 /bin/bash                                
๐Ÿ“š read    ctesting/prod-mysql-0 /bin/bash /etc/bash.bashrc               
๐Ÿ“š read    ctesting/prod-mysql-0 /bin/bash /etc/bash.bashrc               
๐Ÿš€ process ctesting/prod-mysql-0 /bin/bash -ec "password_aux="${MYSQL_ROOT_PASSWORD:-}"

I only see bash running, I don’t see any reads to /etc/shadow because I got a permission denied. The reason for no read message is because I was denied access. Since I got denied, there were no read based events to log. I wonder what happens if we try to write something? 

 % kubectl exec -it -n ctesting  prod-mysql-0 -- /bin/bash
Defaulted container "mysql" out of: mysql, git-sync (init)
I have no name!@prod-mysql-0:/$ echo "blah" /etc/bash.bashrc 
blah /etc/bash.bashrc
I have no name!@prod-mysql-0:/$ echo "blah" >> /etc/bash.bashrc 
bash: /etc/bash.bashrc: Permission denied
I have no name!@prod-mysql-0:/$ ls -al /etc/passwd
-rw-r--r-- 1 root root 922 Apr 18  2023 /etc/passwd
I have no name!@prod-mysql-0:/$ cd /root
bash: cd: /root: Permission denied
I have no name!@prod-mysql-0:/$ cd 
I have no name!@prod-mysql-0:/$ ls -al
total 88
drwxr-xr-x   1 root root 4096 Dec  9 12:54 .
drwxr-xr-x   1 root root 4096 Dec  9 12:54 ..
drwxrwxr-x   2 root root 4096 Oct 29 16:30 .mysqlsh
drwxr-xr-x   2 root root 4096 Oct 29 16:28 bin
drwxr-xr-x   3 root root 4096 Oct 29 16:30 bitnami
drwxr-xr-x   2 root root 4096 Apr 18  2023 boot
drwxr-xr-x   5 root root  360 Dec  9 12:54 dev
drwxrwsrwx   3 root 1001 4096 Dec  9 12:54 docker-entrypoint-initdb.d
drwxr-xr-x   1 root root 4096 Dec  9 12:54 etc
drwxr-xr-x   2 root root 4096 Apr 18  2023 home
drwxr-xr-x   7 root root 4096 Oct 29 16:28 lib
drwxr-xr-x   2 root root 4096 Apr 18  2023 lib64
drwxr-xr-x   2 root root 4096 Apr 18  2023 media
drwxr-xr-x   2 root root 4096 Apr 18  2023 mnt
drwxrwxr-x   1 root root 4096 Dec  9 12:54 opt
dr-xr-xr-x 246 root root    0 Dec  9 12:54 proc
drwx------   2 root root 4096 Apr 18  2023 root
drwxr-xr-x   1 root root 4096 Dec  9 12:54 run
drwxr-xr-x   2 root root 4096 Oct 29 16:28 sbin
drwxr-xr-x   2 root root 4096 Apr 18  2023 srv
dr-xr-xr-x  13 root root    0 Dec  9 12:54 sys
drwxrwxrwt   1 root root 4096 Dec  9 12:54 tmp
drwxrwxr-x  11 root root 4096 Oct 29 16:29 usr
drwxr-xr-x  11 root root 4096 Oct 29 16:29 var
I have no name!@prod-mysql-0:/$ touch .bashrc
touch: cannot touch '.bashrc': Permission denied
I have no name!@prod-mysql-0:/$ cd tmp
I have no name!@prod-mysql-0:/tmp$ touch .bashrc
I have no name!@prod-mysql-0:/tmp$ echo "blah" >> .bashrc 
I have no name!@prod-mysql-0:/tmp$ cat .bashrc 
blah
I have no name!@prod-mysql-0:/tmp$ rm .bashrc 
I have no name!@prod-mysql-0:/tmp$

It looks like I get denied to just about everything, this is a good thing. I was able to write a .bashrc file to /tmp so let’s see what this looks like 

๐Ÿš€ process ctesting/prod-mysql-0 /bin/cat .bashrc                         
๐Ÿ“š read    ctesting/prod-mysql-0 /bin/cat /tmp/.bashrc                    
๐Ÿ“š read    ctesting/prod-mysql-0 /bin/cat /tmp/.bashrc                    
๐Ÿ“š read    ctesting/prod-mysql-0 /bin/cat /tmp/.bashrc                    
๐Ÿ“š read    ctesting/prod-mysql-0 /bin/cat /tmp/.bashrc                    
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /bin/cat .bashrc 0              
๐Ÿš€ process ctesting/prod-mysql-0 /bin/rm .bashrc                          
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /bin/rm .bashrc 0               
๐Ÿ’ฅ exit    ctesting/prod-mysql-0 /bin/bash  0

That’s cool so I see some read operations happening on those files. This is useful to find out if things are being read or written to.

What’s Next?

Checking Out Kubernetes Cluster Network Access Monitoring

Now that I have execution and file access monitoring in place, I’ll want to see what’s going on in the network of my cluster. Before we configure this, we’ll need to gather some details, pod CIDR and service CIDR in use. As mentioned in many of my posts, I’m running on Digital Ocean so I’ll issue the below command to get my pod cidr (The Tetragon instructions mention finding this data in a different location): 

% kubectl get nodes -o json |jq '.items[].metadata.annotations."io.cilium.network.ipv4-pod-cidr"'                  
"blog.shellnetsecurity.com/25"
"blog.shellnetsecurity.com/25"
"blog.shellnetsecurity.com/25"

In order to put these IP addresses to work, I downloaded the template from Tetragon for this policy and added my CIDR to the list 

# output of "gcloud container clusters describe tetragon-benchmarking-oss --zone=us-west2-a | grep -e clusterIpv4Cidr -e servicesIpv4Cidr"
# clusterIpv4Cidr: blog.shellnetsecurity.com/14
#  clusterIpv4Cidr: blog.shellnetsecurity.com/14
#  clusterIpv4CidrBlock: blog.shellnetsecurity.com/14
#  servicesIpv4Cidr: blog.shellnetsecurity.com/20
#  servicesIpv4CidrBlock: blog.shellnetsecurity.com/20
# servicesIpv4Cidr: blog.shellnetsecurity.com/20

# For more information see: https://docs.isovalent.com/user-guide/sec-ops-visibility/workload-identity/index.html#egress-flow-to-suspicious-external-ip
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "monitor-network-activity-outside-cluster-cidr-range"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors:
    - matchArgs:
      - index: 0
        operator: "NotDAddr"
        values:
        - 127.0.0.1
        - blog.shellnetsecurity.com/25
        - blog.shellnetsecurity.com/25
        - blog.shellnetsecurity.com/25

I then applied this to my cluster 

 % kubectl apply -f network_egress_cluster.yaml 
tracingpolicy.cilium.io/monitor-network-activity-outside-cluster-cidr-range created

I then check the events again while I do a curl www.google.com 

๐Ÿš€ process default/nginx-1 /usr/bin/curl www.google.com                   
๐Ÿ”Œ connect default/nginx-1 /usr/bin/curl tcp blog.shellnetsecurity.com:52846 -> 142.250.189.196:80 
๐Ÿ’ฅ exit    default/nginx-1 /usr/bin/curl www.google.com 0

This is working so that’s great!

Filtering Out the Noise in Networking Events

I wasn’t feeling confident that I had the right filters in place. This is supposed to be an egress filter so I want to ignore internal traffic. Let’s get our events again but we’ll grep on connect 

% kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact |grep connect
๐Ÿ”Œ connect wordpress/nginx-int-b8cccd77-npqkn /usr/bin/ssh tcp blog.shellnetsecurity.com:42728 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49110 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-npp-588cc6b9cd-g2x5c /usr/bin/ssh tcp blog.shellnetsecurity.com:33906 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49126 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/nginx-1 /usr/bin/ssh tcp blog.shellnetsecurity.com:35834 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49140 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-int-b8cccd77-npqkn /usr/bin/ssh tcp blog.shellnetsecurity.com:42734 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-npp-588cc6b9cd-g2x5c /usr/bin/ssh tcp blog.shellnetsecurity.com:33918 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49150 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/nginx-1 /usr/bin/ssh tcp blog.shellnetsecurity.com:35846 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49152 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-int-b8cccd77-npqkn /usr/bin/ssh tcp blog.shellnetsecurity.com:42748 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-npp-588cc6b9cd-g2x5c /usr/bin/ssh tcp blog.shellnetsecurity.com:33928 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49160 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/nginx-1 /usr/bin/ssh tcp blog.shellnetsecurity.com:35848 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49174 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-int-b8cccd77-npqkn /usr/bin/ssh tcp blog.shellnetsecurity.com:42764 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49188 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-npp-588cc6b9cd-g2x5c /usr/bin/ssh tcp blog.shellnetsecurity.com:33930 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/wordpress-0 /opt/bitnami/php/sbin/php-fpm tcp blog.shellnetsecurity.com:41246 -> blog.shellnetsecurity.com:3306 
๐Ÿ”Œ connect wordpress/wordpress-0 /opt/bitnami/php/sbin/php-fpm tcp blog.shellnetsecurity.com:41250 -> blog.shellnetsecurity.com:3306 
๐Ÿ”Œ connect default/nginx-1 /usr/bin/ssh tcp blog.shellnetsecurity.com:54866 -> 192.30.255.113:22

It looks like I’ve got some traffic showing up that’s internal to my cluster going to blog.shellnetsecurity.com so I’ll want to ignore those events. I know these are coming from my service IPs so I’ll do a quick hack for it 

% kubectl get svc -A |awk '{print $4}'
CLUSTER-IP
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
None
blog.shellnetsecurity.com
None
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
None
None
None
None
None
None
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com
blog.shellnetsecurity.com

From a quick look, I see this spanning blog.shellnetsecurity.com/16 so I’ll just add that to my YAML: 

# output of "gcloud container clusters describe tetragon-benchmarking-oss --zone=us-west2-a | grep -e clusterIpv4Cidr -e servicesIpv4Cidr"
# clusterIpv4Cidr: blog.shellnetsecurity.com/14
#  clusterIpv4Cidr: blog.shellnetsecurity.com/14
#  clusterIpv4CidrBlock: blog.shellnetsecurity.com/14
#  servicesIpv4Cidr: blog.shellnetsecurity.com/20
#  servicesIpv4CidrBlock: blog.shellnetsecurity.com/20
# servicesIpv4Cidr: blog.shellnetsecurity.com/20

# For more information see: https://docs.isovalent.com/user-guide/sec-ops-visibility/workload-identity/index.html#egress-flow-to-suspicious-external-ip
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "monitor-network-activity-outside-cluster-cidr-range"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors:
    - matchArgs:
      - index: 0
        operator: "NotDAddr"
        values:
        - 127.0.0.1
        - blog.shellnetsecurity.com/25
        - blog.shellnetsecurity.com/25
        - blog.shellnetsecurity.com/25

After editing that, I do another apply: 

% kubectl apply -f network_egress_cluster.yaml                                                                        
tracingpolicy.cilium.io/monitor-network-activity-outside-cluster-cidr-range configured

Let’s check the events again: 

% kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact |grep connect
๐Ÿ”Œ connect wordpress/nginx-int-b8cccd77-npqkn /usr/bin/ssh tcp blog.shellnetsecurity.com:53492 -> 192.30.255.112:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:51674 -> 192.30.255.112:22 
๐Ÿ”Œ connect default/nginx-1 /usr/bin/ssh tcp blog.shellnetsecurity.com:46870 -> 192.30.255.112:22 
๐Ÿ”Œ connect wordpress/nginx-npp-588cc6b9cd-g2x5c /usr/bin/ssh tcp blog.shellnetsecurity.com:42074 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:51676 -> 192.30.255.112:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:35328 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-int-b8cccd77-npqkn /usr/bin/ssh tcp blog.shellnetsecurity.com:53500 -> 192.30.255.112:22 
๐Ÿ”Œ connect default/nginx-1 /usr/bin/ssh tcp blog.shellnetsecurity.com:46884 -> 192.30.255.112:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:35334 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-npp-588cc6b9cd-g2x5c /usr/bin/ssh tcp blog.shellnetsecurity.com:33802 -> 192.30.255.112:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:51680 -> 192.30.255.112:22 
๐Ÿ”Œ connect wordpress/nginx-int-b8cccd77-npqkn /usr/bin/ssh tcp blog.shellnetsecurity.com:53506 -> 192.30.255.112:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:51692 -> 192.30.255.112:22 
๐Ÿ”Œ connect default/nginx-1 /usr/bin/ssh tcp blog.shellnetsecurity.com:47142 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49710 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-npp-588cc6b9cd-g2x5c /usr/bin/ssh tcp blog.shellnetsecurity.com:44334 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:35322 -> 192.30.255.112:22 
๐Ÿ”Œ connect wordpress/nginx-int-b8cccd77-npqkn /usr/bin/ssh tcp blog.shellnetsecurity.com:34810 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/nginx-1 /usr/bin/ssh tcp blog.shellnetsecurity.com:38146 -> 192.30.255.113:22 
๐Ÿ”Œ connect kube-system/do-node-agent-m2h2g /bin/do-agent tcp 206.189.208.210:50788 -> 169.254.169.254:80 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49722 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-npp-588cc6b9cd-g2x5c /usr/bin/ssh tcp blog.shellnetsecurity.com:44344 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49732 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-int-b8cccd77-npqkn /usr/bin/ssh tcp blog.shellnetsecurity.com:34820 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/nginx-1 /usr/bin/ssh tcp blog.shellnetsecurity.com:38158 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49740 -> 192.30.255.113:22 
๐Ÿ”Œ connect wordpress/nginx-npp-588cc6b9cd-g2x5c /usr/bin/ssh tcp blog.shellnetsecurity.com:44350 -> 192.30.255.113:22 
๐Ÿ”Œ connect default/log-gen-59f94c9d86-h299t /usr/bin/ssh tcp blog.shellnetsecurity.com:49744 -> 192.30.255.113:22

Ok so this looks pretty good.

Wrapping Up

Ok so this blog post was just supposed to be a quick introduction into runtime security observation and I realize that it got away from me ๐Ÿ™‚

It looks like I’ve got some events coming through into the system but I think I need to revisit tuning these events. I’ll create another blog post that explains some additional tuning for this beast.

Related posts:

  1. Enhancing Kubernetes Security and Compliance with Starboard Audit Reports: A Practical Guide
  2. How to Install and Use Starboard to Protect Your Kubernetes Cluster
  3. Leveraging InitContainers to Streamline Bitnami MySQL Initialization
  4. Kubernetes Upgrades Break My DigitalOcean LoadBalancer