ungleich-k8s/v3-calico

Introduction

The following guide shows how to setup an IPv6 only cluster at ungleich.

Steps

• Boot Alpine
• Configure with cdist to get cri-o configured

Control plane

Initialise with all components:

kubeadm init -service-cidr 2a0a:e5c0:13:aaa::/108 --pod-network-cidr 2a0a:e5c0:13:bbb::/64

We cannot yet skip kube-proxy, because calico does not support eBPF for IPv6. Cilium supports IPv6 eBPF, but on the other hand does not support automatic BGP peering. So the following does not work:

kubeadm init --skip-phases=addon/kube-proxy --service-cidr 2a0a:e5c0:13:aaa::/108 --pod-network-cidr 2a0a:e5c0:13:bbb::/64

Alpine / kubelet hack

Due to some misconfiguration on alpine, DURING the kubeadm init we need to modify the generated /var/lib/kubelet/config.yaml to replace "cgroupDriver: systemd" with "cgroupDriver: cgroupfs".

The same is necessary on the worker nodes, however that can be done anytime before you plan to schedule containers on them, after the kubeadm join request.

Worker nodes

kubeadm join [2a0a:e5c0:13:0:225:b3ff:fe20:38cc]:6443 --token bw3x98.chp31kcgcd4b5fpf --discovery-token-ca-cert-hash sha256:...

CNI/networking

kubectl apply -f calico.yaml

Warning: needs to be updated:

serviceaccount/calico-kube-controllers created
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/calico-kube-controllers created

Checking pods:

[21:53] server47.place7:~/v3-calico# kubectl -n kube-system get pods
NAME                                       READY   STATUS              RESTARTS   AGE
calico-kube-controllers-6d8ccdbf46-4xzz9   0/1     Pending             0          60s
calico-node-5gkp9                          0/1     Init:0/3            0          60s
calico-node-8lct9                          0/1     Init:0/3            0          60s
calico-node-jmjhn                          0/1     Init:0/3            0          60s
calico-node-krnzr                          0/1     Init:ErrImagePull   0          60s
coredns-558bd4d5db-4rvrf                   0/1     Pending             0          3m40s
coredns-558bd4d5db-g9lbx                   0/1     Pending             0          3m40s
etcd-server47                              1/1     Running             0          3m56s
kube-apiserver-server47                    1/1     Running             0          3m55s
kube-controller-manager-server47           1/1     Running             0          3m56s
kube-scheduler-server47                    1/1     Running             0          3m55s
[21:54] server47.place7:~/v3-calico#

Getting calicoctl

kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml

And alias it:

alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"

Configuring BGP routing

calicoctl create -f - < bgp....yaml

Setup a test deployment

Do NOT use https://k8s.io/examples/application/deployment.yaml. It contains an outdated nginx container that has no IPv6 listener. You will get results such as

[19:03] server47.place7:~/ungleich-k8s/v3-calico#  curl http://[2a0a:e5c0:13:bbb:176b:eaa6:6d47:1c41]
curl: (7) Failed to connect to 2a0a:e5c0:13:bbb:176b:eaa6:6d47:1c41 port 80: Connection refused

if you use that deployment. Instead use something on the line of the included nginx-test-deployment.yaml:

kubectl apply -f nginx-test-deployment.yaml