Disclosure: I have included some affiliate / referral links in this post. There’s no cost to you for accessing these links but I do indeed receive some incentive for it if you buy through them.
I’ve talked about it in previous posts about my thus far overall enjoyment running in DigitalOcean. While I had tinkered with a number of other cloud providers, I settled with them for many things. I do still run in some other providers like OVHCloud (maybe more on my project there for another day). Despite my love for DigitalOcean, I do have one complaint regarding their Kubernetes and their LoadBalancer.
The Problem: I’m Cheap
I guess thrifty sounds so much better but I’m cheap. It’s a fact. I have in fact created my own problem with DigitalOcean due to my cheapness. They do have a number of excellent integration points between their Kubernetes and other components such as storage and load balancers. I can issue Kubernetes commands to create a new LoadBalancer or PVC and boom life is good. My problem is that LoadBalancers cost money. To date, I have only been able to figure out a 1:1 mapping between the LoadBalancer and Kubernetes. This 1:1 means that I can only manage a single LoadBalancer per exposed port.
If I only ever intend to expose a single application to the world, this is great! This is not me. I run a number of different applications that I want to expose. That means I need to pay for a LoadBalancer for each application or do I? Here come the Forwarding Rules! Each LoadBalancer can be configured with a number of forwarding rules like so:
With these rules in place, I’m able to expose multiple ports/applications on the same load balancer. This is wonderful except upgrades to the Kubernetes clusters like to blow away my custom settings such as:
- Forwarding Rules
- SSL Redirects
- Proxy Protocol
- Backend Keepalive
For the longest time, I had to come back in reconfigure everything every time I did a Kubernetes cluster upgrade. Worse yet, I didn’t know things got blown away whenever the cluster upgraded automatically. I had setup port/application monitors to alert me when things when down so I could manually reconfigure them.
The Solution : DigitalOcean API
While the manual fix has always been a waste of time and has sometimes prevented me from upgrading Kubernetes (bad security d00d), I still did the upgrades and manually fixed it. I never really learned a “new thing” to try and get this fixed in a less manual manner. Today was the the day I changed all of that. I’m sure there’s some other way that I hadn’t thought of yet but we’re going with baby steps. Instead of taking manual screenshots of the configuration page for the LoadBalancer and then trying to manually go back in and change the settings to what I thought they were, I am now using the API. Some good general documentation on the DigitalOcean API can be found here:
Setting Up API Access
The first step in getting all of this working is getting an API token. There’s no sense reinventing the wheel when DigitalOcean already has something written up very well such as How to Create a Personal Access Token. This walks you through creating the token to be used with the API. It is very important to make sure you create a token that has write access.
Get Your Existing LoadBalancer Configuration
With token in hand, it is time to get a copy of your existing configuration with a curl command. First, you need to know the id of your load balancer so we just query for all load balancers:
% curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer your_api_token_here" https://api.digitalocean.com/v2/load_balancers|jq .
{
"load_balancers": [
{
"id": "ffff-ffff-ffff-ffff-b75c",
"name": "my-lb-01",
"size": "lb-small",
"algorithm": "round_robin",
"status": "active",
"created_at": "2019-10-25T19:56:00Z",
"forwarding_rules": [
{
"entry_protocol": "tcp",
"entry_port": 80,
"target_protocol": "tcp",
"target_port": 31640,
"certificate_id": "",
"tls_passthrough": false
},
{
"entry_protocol": "tcp",
"entry_port": 4514,
"target_protocol": "tcp",
"target_port": 31643,
"certificate_id": "",
"tls_passthrough": false
}
],
"region": {
"name": "San Francisco 2",
"slug": "sfo2"
},
"tag": "",
"droplet_ids": [
111,
222,
333
],
"redirect_http_to_https": false,
"enable_proxy_protocol": false,
"enable_backend_keepalive": false,
},
{
"id": "ffff-ffff-ffff-ffff-72a4",
"name": "my-lb-02",
"size": "lb-small",
"algorithm": "round_robin",
"status": "active",
"created_at": "2020-12-02T07:54:13Z",
"forwarding_rules": [
{
"entry_protocol": "https",
"entry_port": 443,
"target_protocol": "http",
"target_port": 31645,
"certificate_id": "aaaa-aaaa-aaaa-aaaa-bcf8",
"tls_passthrough": false
},
{
"entry_protocol": "tcp",
"entry_port": 80,
"target_protocol": "tcp",
"target_port": 31645,
"certificate_id": "",
"tls_passthrough": false
}
],
"region": {
"name": "San Francisco 2",
"slug": "sfo2"
},
"tag": "",
"droplet_ids": [
111,
222,
333
],
"redirect_http_to_https": true,
"enable_proxy_protocol": true,
"enable_backend_keepalive": true,
}
],
"links": {},
"meta": {
"total": 2
}
}There are two load balancers here in this example, my-lb-01 and my-lb-02. While my-lb-01 was my original load balancer that gave me the most trouble, I’m going to focus on my-lb-02 since it has more customizations not just to the forwarding rules.
We need to first identify the configuration that we’d like to save. Then, we’ll save this configuration into it’s own json, let’s call it my-lb-02.json. Notice in the above JSON, the configurations are housed within a “loadbalancer” array? In order to create our my-lb-02.json file, we simply pull the single JSON element from the array like this:
{
"id": "ffff-ffff-ffff-ffff-72a4",
"name": "my-lb-02",
"size": "lb-small",
"algorithm": "round_robin",
"status": "active",
"created_at": "2020-12-02T07:54:13Z",
"forwarding_rules": [
{
"entry_protocol": "https",
"entry_port": 443,
"target_protocol": "http",
"target_port": 31645,
"certificate_id": "aaaa-aaaa-aaaa-aaaa-bcf8",
"tls_passthrough": false
},
{
"entry_protocol": "tcp",
"entry_port": 80,
"target_protocol": "tcp",
"target_port": 31645,
"certificate_id": "",
"tls_passthrough": false
}
],
"region": {
"name": "San Francisco 2",
"slug": "sfo2"
},
"tag": "",
"droplet_ids": [
111,
222,
333
],
"redirect_http_to_https": true,
"enable_proxy_protocol": true,
"enable_backend_keepalive": true,
}We need to remove a few useless items from that JSON so remove the following:
- status
- name
- size
- created_at
- region (do remember the “slug” entry as we’ll need this to recreate the region)
As noted above, we also need to remove the existing region entry and instead replace it with the value of “slug” aka “sfo2” in this example. With those changes made, here’s our new JSON:
{
"id": "ffff-ffff-ffff-ffff-72a4",
"algorithm": "round_robin",
"forwarding_rules": [
{
"entry_protocol": "https",
"entry_port": 443,
"target_protocol": "http",
"target_port": 31645,
"certificate_id": "aaaa-aaaa-aaaa-aaaa-bcf8",
"tls_passthrough": false
},
{
"entry_protocol": "tcp",
"entry_port": 80,
"target_protocol": "tcp",
"target_port": 31645,
"certificate_id": "",
"tls_passthrough": false
}
],
"region": "sfo2",
"tag": "",
"droplet_ids": [
111,
222,
333
],
"redirect_http_to_https": true,
"enable_proxy_protocol": true,
"enable_backend_keepalive": true,
}How Do I Unbreak Things in the Future?
I’m glad you asked! Now that you have your my-lb-02 JSON file ready to go, you can simply wait for the next upgrade of your Kubernetes cluster to rebuild everything. Below, you can see my-lb-02 broken in the DigitalOcean control panel:
There’s one little catch to fixing everything. You’ll need to first get the IDs of the new cluster nodes in order to be able to add them to the load balancer. Whenever the cluster is upgraded, DigitalOcean deletes the old versioned node and adds in a new versioned one. You can do this by doing a GET to one of the load balancer’s configurations:
% curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer your_api_token_here" https://api.digitalocean.com/v2/load_balancers/ffff-ffff-ffff-ffff-72a4|jq .load_balancer.droplet_ids
[
123,
456,
789
] In order to make my life easier, I piped my results through jq and told it to only bring back the json path I cared about, load_balancer.droplet_ids. Now we see that the droplets have changed from our original list of 111, 222, 333 to 123, 456, 789. We need to make this change to our JSON
{
"id": "ffff-ffff-ffff-ffff-72a4",
"algorithm": "round_robin",
"forwarding_rules": [
{
"entry_protocol": "https",
"entry_port": 443,
"target_protocol": "http",
"target_port": 31645,
"certificate_id": "aaaa-aaaa-aaaa-aaaa-bcf8",
"tls_passthrough": false
},
{
"entry_protocol": "tcp",
"entry_port": 80,
"target_protocol": "tcp",
"target_port": 31645,
"certificate_id": "",
"tls_passthrough": false
}
],
"region": "sfo2",
"tag": "",
"droplet_ids": [
123,
456,
789
],
"redirect_http_to_https": true,
"enable_proxy_protocol": true,
"enable_backend_keepalive": true,
}With the JSON updated, we now issue a PUT command to the load balancer API for the specific load balancer like so:
% curl -X PUT -H "Content-Type: application/json" -H "Authorization: Bearer your_api_token_here" https://api.digitalocean.com/v2/load_balancers/ffff-ffff-ffff-ffff-72a4 -d @my-lb-02.jsonNow we can go look at the control panel again and confirm everything is back to normal!
Everything Works Great!
After an upgrade runs, I can simply come back through with a few minor steps and put everything back the way it should. Yes, there’s still some manual aspects to this and automating all of it shouldn’t be too terrible but I’ll save that for another time when I decide that this manual process is just too much anymore. Although, it took me nearly a year to hate the original manual process….