How to setup High Availability K3s Cluster

Overview

Hi there Kubernetes explorers and ‘home labers’. Here i want to document the steps how to setup high availability k3s cluster with kube-vip. This setup will enable us to have an HA kubernetes cluster without the need of adding an external load balancer. I want to avoid of wasting as much of resources as possible and allocate most of the resources to the cluster, this is the way - for me at least :). This setup will be created on VMs, running on top of Proxmox cluster.

Requirements:

• Three control plane VM nodes,
• Two or more worker node VMs,
• kube-vip service,
• sudo or root access to all the nodes,
• static ip addresses on all the nodes,
• whitelisted ports on firewall: 6443, 10250, 2379-2380 TCP.
• all VMs should have at least 2 x CPU cores, 4 GB of RAM and minimum of 10gb HDD.

Why K3s?

Well the first and obvious choice was that it’s really lightweight? How lightweight? All core binaries are less than 100mb. Second reason is that it’s optimized in a way that it does need not much hardware to run a kubernetes cluster. So if you’re like me and have couple of old mini PCs running Proxmox with couple of VMs, this for me is an excellent choice.

Another good reason why k3s is that installation process is simple and it comes packaged in with a lot of great core services, such as - CoreDNS, Flannel CNI, Traefik ingress controller, containerd container runtime, etc. All of these components can be replaced and you can even opt out for these not to be installed when you run the setup script and install the service of your choosing afterwards.

So, once you install it, you can start using it right away.

To summarize the k3s choice - Small footprint, ease of installation, reduced overhead and optimized for lower end hardware.

Why three control plane nodes?

Three control planes are required due to the core nature of Kubernetes and it’s database - etcd. To maintain quorum and ensure database persistence and redundancy during maintenance or downtime, three control planes are necessary. This approach is recommended when using an embedded database because it eliminates the need for an external dependency. For me this meant one less dependency to worry about.

If you can’t afford three control planes, you can use 2 or 1 control plane node, but it would be advisable in this setup to use external DB storage backend such as MySQL or Postgres(meaning to run these as a separate VMs/instances). If you decide to go with 1 control plane and etcd for the k8s database, in this setup is advisable to at least have etcd snapshots/backups.

Alternative solution with external db storage backend

For this process it will be best to reffer with the k3s documentation on this topic - LINK. But the process seems straightforward too. When executing the server installation, you need to add the --datastore-endpoint option and add as a value the remote DB engine parameters and credentials.

Why kube-vip?

Kube-vip is a load balancer for kubernetes that supports various different network modes(Layer2 ARP, BGP etc) and i need it for Layer2 mode and ARP(to promote and assign a virtual IP on the same existing network interface on the VM) and i will use it to be load balancer for the control planes, to be more precise, it will be the load balancer for the kubernetes API.

Another great benefit and feature of kube-vip is that i can deploy it as a daemonset(which i did) and it comes with failover capabilities, so if one of the daemonsets pods goes down, it can assign a different daemonset pod on a different node to be primary load balancer pod with the same IP address. And this setup will ensure for me to have consistent access to the Kubernetes API.

Alternative solution to this is to add an external load balancer such as Nginx, Traefik, etc outside of the cluste and run it as a separate instance.

But for me again this means less resources to expend for an external load balancer.

Alternative solution with external load balancer

Another approach to add the Kubernetes API load balancer is to provision external load balancer, such as Nginx or similiar. Meaning the load balancer must be outside of the kubernetes cluster as separate instance. And when joining the server nodes, we’ll need in the k3s install server command define the first node as the server and in the --tls-san option, add the Nginx IP address as the value when joining servers but when joining agents, in the server option add Nging IP address as the value.

And the Nginx configuration should look something like this:

# Uncomment this next line if you are NOT running nginx in docker
# By loading the ngx_stream_module module, Nginx can handle TCP/UDP traffic in addition to HTTP traffic
load_module /usr/lib/nginx/modules/ngx_stream_module.so;
events {}

stream {
  upstream k3s_servers {
    server <IP1>:6443; # Change to the IP of the K3s first master VM
    server <IP2>:6443; # Change to the IP of the K3s second master VM
    server <IP3>:6443; # Change to the IP of the K3s third master VM
  }

  server {
    listen 6443;
    proxy_pass k3s_servers;
  }
}

1. Kube-vip setup

Before we start with the k3s installation, the kube-vip setup is a must have requirement! The reason is that we plan to deploy and run it as a daemonset component(meaning it acts as an agent pod and soon as new control plane is added to the cluster, it will deploy another agent pod to the new node) and for this we need several things - create neccesary directory, kube-vip RBAC and kube-vip daemonset yaml file. Let’s start!

1a. Add kube-vip RBAC on the first node

Create the directories that will have this path /var/lib/rancher/k3s/server/manifests/ You can do it easily with the following command:

mkdir -p /var/lib/rancher/k3s/server/manifests/

In the same directory we need to download the RBAC for kube-vip from the kube-vip website and we can do that with this command:

curl https://kube-vip.io/manifests/rbac.yaml > /var/lib/rancher/k3s/server/manifests/kube-vip-rbac.yaml

Once RBAC is downloaded, the final requirement for kube-vip is to add in the same directory the daemonset yaml file. In the same directory either create a new yaml file or just append in the kube-vip-rbac.yaml the following kube-vip example manifest for ARP mode(i used the same for my k3s cluster):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  creationTimestamp: null
  name: kube-vip-ds
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: kube-vip-ds
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: kube-vip-ds
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: Exists
            - matchExpressions:
              - key: node-role.kubernetes.io/control-plane
                operator: Exists
      containers:
      - args:
        - manager
        env:
        - name: vip_arp
          value: "true"
        - name: port
          value: "6443"
        - name: vip_interface
          value: eth0
        - name: vip_cidr
          value: "32"
        - name: cp_enable
          value: "true"
        - name: cp_namespace
          value: kube-system
        - name: vip_ddns
          value: "false"
        - name: svc_enable
          value: "true"
        - name: vip_leaderelection
          value: "true"
        - name: vip_leaseduration
          value: "5"
        - name: vip_renewdeadline
          value: "3"
        - name: vip_retryperiod
          value: "1"
        - name: address
          value: 192.168.0.249
        image: ghcr.io/kube-vip/kube-vip:v0.4.0
        imagePullPolicy: Always
        name: kube-vip
        resources: {}
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - NET_RAW
            - SYS_TIME
      hostNetwork: true
      serviceAccountName: kube-vip
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
  updateStrategy: {}
status:
  currentNumberScheduled: 0
  desiredNumberScheduled: 0
  numberMisscheduled: 0
  numberReady: 0

Here’s important to change and adjust the env values for address and for the vip_interface per your VM and network setup. Also it’s important to note that the vip_interface name must be either same on all VMs or add the list of interfaces that all VMs have.

  - name: vip_interface
    value: eth0
  
  - name: address
    value: 192.168.0.249

This is it for the kube-vip. Now we can move on with the k3s installation.

2. Setup High Availability K3s Cluster

2a. Install and initialize cluster on the first node

To install k3s and initialize cluster with all the goodies and batteries, we just need a one liner command from the k3s documentation that will execute the k3s install script.

curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --cluster-init --tls-san=kube-vip-ip-address

Here’s important to note - the value of --tls-san option must be the IP address you assigned for the kube-vip service. The command must contain the server option since that will tell the script to install the server/control planes binaries of the k3s. Also for the K3S_TOKEN=SECRET, replace the SECRET with either a randomly generated string, or a passphrase or something similar. This SECRET is also important and it’s value will be added as a suffix for the server token and the same token we need to use in order to join other nodes to the cluster.

After a few minutes, k3s on the first node should be installed, up and running with the kube-vip service too. The easiest way to double check is with kubectl commands:

kubectl get nodes
kubectl get pod --all-namespaces

And the result should look like this:

Here you should see kube-vip pod running, and this means we’re able to install kube-vip successfully. And to confirm if kube-vip was able to acquire the IP addresss you defined and assign to the NIC interfaces, double check it with ip a command and the result should look like in the picture bellow:

In this picture we see on the same NIC interface the node/VM ip address and the kube-vip address. We can move on with adding more nodes to the cluster.

2b. Join other control plane nodes to the cluster

Now we can join other nodes to the cluster. First we need to exract the value of the SERVER TOKEN from the first node, write it down. We will use this token so that we can add other nodes to the cluster.

cat /var/lib/rancher/k3s/server/token

The output of the token should look something like this:

K052cef3cacc7643811b76d5ef40d90c1df3927cbb10f672362a1437e108a84d460::server:SECRET-YOU-ENTERED-ON-K3S-INSTALL

Copy or write down the entire token value.

Join the second and third control node with the following command:

curl -sfL https://get.k3s.io | K3S_TOKEN=FULL_TOKEN::server:SECRET-YOU-ENTERED-ON-K3S-INSTALL sh -s - server --server https://kube-vip-ip-address:6443

If by any reason the join process fails and you get connection refused error, time outs, no certs or similar, try joining the cluster with this command:

curl -sfL https://get.k3s.io | K3S_TOKEN=FULL_TOKEN::server:SECRET-YOU-ENTERED-ON-K3S-INSTALL sh -s - server --server https://ip-address-of-the-first-node:6443 --tls-san=kube-vip-ip-address

Again double check with kubectl get nodes if all the nodes have joined the cluster.

The end result should look like this, all three nodes joined and ready. We can start adding agent nodes to the cluster.

2c. Join agent nodes to the cluster

Last step is to join the agent nodes or worker nodes to the cluster. The process for them is the same and we use the same token we used for server nodes.

Command to join the agent nodes:

curl -sfL https://get.k3s.io | K3S_TOKEN=FULL_TOKEN::server:SECRET-YOU-ENTERED-ON-K3S-INSTALL sh -s - agent --server https://kube-vip-ip-address:6443

Small but an important difference here is to specify in the command the agent option in order to install the agent binaries.

Double check if the agent nodes joined successfully and are ready.

Eureka! Great success!

Summary

After all the steps, we now have an HA k3s cluster. Let’s recap the process on how to setup high availability k3s cluster with kube-vip and what we did here:

• Prepared VMs with static IPs and whitelisted ports on firewall for kubernetes communication,
• Downloaded and prepared the kube-vip configuration to install it as daemonset on k3s,
• Installed and initiliazed k3s cluster on the first,
• Added to the cluster other two control plane nodes,
• Added agent nodes to the cluster.

Thank you for your time, take care and happy ’labing’.