Nowadays it is undeniable there are many ways to bring up a Kubernetes cluster. In the case you decide to run locally, options such as minikube or kind are very popular. Nevertheless, they lack of automation. At work I use terraform, and when I came across with terraform-provider-libvirt I had the idea of creating k8s clusters. Once terraform is applied I just need to run kubeadm init and join from the worker nodes, and finally install any of the network addons.
QEMU KVM + Terraform
I assume both qemu-kvm and terraform is already installed. First step requries to download the terraform-provider-libvirt plugin to ~/.terraform.d/plugins. The are releases depending on your linux distribution. A simple test to verify everything is working, create main.tf with just the provider definition.
provider "libvirt" {
uri = "qemu:///system"
}
If you run $ terraform init && terraform plan you probably see the message
No changes. Infrastructure is up-to-date.
The only issue I found was related to permissions when creating the volumes in /var/lib/libvirt/images. Long story short, there was already a thread on Github and later added to the documentation. The option to set in /etc/libvirt/qemu.conf is security_driver = "none". Otherwise SELinux gets enforced even if it was disabled at system level. After the previous change, everything worked out.
Kubernetes cluster
As I mentioned above, The installation of Kubernetes is done with kubeadm, the documentation improved over the last months and makes it much easier. The idea idea behind of using terraform is to have the required building blocks, KVM instances in this case before using kubeadm. For that reason, I created a terraform module. The key was to use a cloudinit configuration and automate as many steps as possible.
Below is a un extract of the module, the idea is to have
#
# worker nodes
#
resource "libvirt_volume" "worker_image" {
count = local.workers
name = "worker-${count.index + 1}"
pool = "default"
source = "https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img"
format = "qcow2"
}
resource "libvirt_volume" "worker" {
count = local.workers
name = "volume-${count.index + 1}"
base_volume_id = element(libvirt_volume.worker_image.*.id, count.index)
size = var.worker_root_size
}
resource "libvirt_cloudinit_disk" "commoninit" {
count = local.workers
name = "k8s-worker-${count.index + 1}-commoninit.iso"
pool = "default"
user_data = element(data.template_file.user_data.*.rendered, count.index + 1)
}
data "template_file" "user_data" {
count = local.workers
template = file("${path.module}/kubernetes_cloudinit.cfg")
vars = {
hostname = "k8s-worker${count.index + 1}"
public_key = var.public_key
k8s_version = var.k8s_version
k8s_minor = var.k8s_minor
}
}
resource "libvirt_domain" "worker" {
count = local.workers
name = "k8s-worker${count.index + 1}"
memory = var.worker_mem
vcpu = var.worker_cpu
disk {
volume_id = element(libvirt_volume.worker.*.id, count.index + 1)
}
network_interface {
network_name = "default"
}
cloudinit = element(libvirt_cloudinit_disk.commoninit.*.id, count.index + 1)
console {
type = "pty"
target_port = "0"
target_type = "serial"
}
graphics {
type = "spice"
listen_type = "address"
autoport = "true"
}
autostart = var.enable_autostart
}
terraform {
required_version = ">= 0.12"
}
The are some variables to allow definining more than 1 worker node. The interesting part is the kubernetes_cloudinit.cfg.
#cloud-config
hostname: ${hostname}
users:
- name: sgm
sudo: ALL=(ALL) NOPASSWD:ALL
groups: users, admin
home: /home/sgm
shell: /bin/bash
lock_passwd: false
ssh-authorized-keys:
- ${public_key}
- name: ubuntu
sudo: ALL=(ALL) NOPASSWD:ALL
groups: users, admin
home: /home/ubuntu
shell: /bin/bash
lock_passwd: false
disable_root: false
chpasswd:
list: |
ubuntu:ubuntu
expire: False
runcmd:
- echo "AllowUsers sgm" >> /etc/ssh/sshd_config
- systemctl restart sshd
packages:
- apt-transport-https
- ca-certificates
- curl
- software-properties-common
- gnupg2
apt_sources:
- source: "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
keyid: 0EBFCD88
filename: docker-ce.list
- source: "deb https://apt.kubernetes.io/ kubernetes-xenial main"
filename: kubernetes.list
keyid: BA07F4FB
packages:
- qemu-guest-agent
- containerd.io=1.2.13-2
- docker-ce=5:19.03.12~3-0~ubuntu-bionic
- docker-ce-cli=5:19.03.12~3-0~ubuntu-bionic
- kubeadm=${k8s_version}.${k8s_minor}-00
- kubectl=${k8s_version}.${k8s_minor}-00
- kubelet=${k8s_version}.${k8s_minor}-00
write_files:
- path: /etc/docker/daemon.json
content: |
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
- path: /etc/sysctl.d/k8s.conf
content: |
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
runcmd:
- mkdir -p /etc/systemd/system/docker.service.d
- gpasswd -a sgm docker
- systemctl daemon-reload
- systemctl restart docker
- systctl --system
A nice to have is the kubernetes version, said and one. Now the only thing I need to know is define a cluster.
module "k8s-testing" {
source = "/home/sgm/git/terraform/kvm/k8s"
public_key = file("/home/sgm/.ssh/id_libvirt.pub")
masters_count = 1
master_mem = 2048
master_cpu = 2
workers_count = 40
worker_mem = 1024
worker_cpu = 1
k8s_version = "1.18"
k8s_minor = "4"
enable_autostart = true
}
output "ips" {
value = module.k8s-testing.ips
}
After running terraform apply you can verifying that instances are up. In case you need a crash course in virsh commands.
sudo virsh list
Id Name State
-----------------------------
1 k8s-master1 running
2 k8s-worker3 running
3 k8s-worker1 running
4 k8s-worker2 running
Once the cloudinit ends, you can reach the console or ssh with a public key. It is possible to use terraform refresh to get the IPs. Another option is to use virsh:
sudo virsh net-dhcp-leases default
Expiry Time MAC address Protocol IP address Hostname Client ID or DUID
------------------------------------------------------------------------------------------------------------------------------------------------
2020-08-20 23:11:16 52:54:00:0e:b9:9b ipv4 192.168.122.188/24 k8s-w3 ff:b5:5e:67:ff:00:02:00:00🆎11:b2:f8:a8:e4:98:ee:21:2d
2020-08-20 23:11:16 52:54:00:46:bb:bf ipv4 192.168.122.246/24 k8s-w2 ff:b5:5e:67:ff:00:02:00:00🆎11:35:b7:a3:5f:26:08:8d:ee
2020-08-20 23:11:16 52:54:00:aa:45:24 ipv4 192.168.122.196/24 k8s-w1 ff:b5:5e:67:ff:00:02:00:00🆎11:b6:51:cc:ea:8e:8c:e2:48
2020-08-20 23:11:16 52:54:00:86:73:68 ipv4 192.168.122.4/24 k8s-m1 ff:b5:5e:67:ff:00:02:00:00🆎11:1d:ab:55:19:f9:2f:bc:00
If you try to connect using virsh console k8s-m1 you may see the cloudunit running, this is also great way to troubleshooting.
Connected to domain k8s-master1
Escape character is ^]
Ubuntu 18.04.5 LTS k8s-master1 ttyS0
k8s-m1 login: [ 42.227031] cloud-init[969]: Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
[ 42.386846] cloud-init[969]: Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
[ 42.455497] cloud-init[969]: Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
[ 42.531884] cloud-init[969]: Get:4 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
[ 42.968395] cloud-init[969]: Get:6 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages [12.5 kB]
[ 43.227532] cloud-init[969]: Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [808 kB]
[ 43.299882] cloud-init[969]: Get:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
[ 43.667188] cloud-init[969]: Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [8570 kB]
[ 43.946882] cloud-init[969]: Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8993 B]
[ 44.209239] cloud-init[969]: Get:10 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [38.8 kB]
[ 44.281594] cloud-init[969]: Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [689 kB]
[ 44.416072] cloud-init[969]: Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [228 kB]
[ 44.526903] cloud-init[969]: Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [8112 B]
[ 44.529225] cloud-init[969]: Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2852 B]
[ 45.282212] cloud-init[969]: Get:15 http://archive.ubuntu.com/ubuntu bionic/universe Translation-en [4941 kB]
[ 45.695633] cloud-init[969]: Get:16 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [151 kB]
[ 45.701082] cloud-init[969]: Get:17 http://archive.ubuntu.com/ubuntu bionic/multiverse Translation-en [108 kB]
[ 45.769814] cloud-init[969]: Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1032 kB]
[ 45.806097] cloud-init[969]: Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1097 kB]
[ 45.916802] cloud-init[969]: Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [342 kB]
[ 45.927765] cloud-init[969]: Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [19.2 kB]
In the end I quite happy with the setup, the kvm instances spin up quickly, in case you decide to set autostart=false a terraform apply in the next boot will turn them on.