## Introduction The following guide shows how to setup an IPv6 only cluster at ungleich. ## Steps - Boot Alpine - Configure with cdist to get cri-o configured ## Control plane Initialise with all components: ``` kubeadm init --config kubeadm-config.yaml ``` We need to specify the **--config** option to inject correct cgroupdriver! We cannot yet skip kube-proxy, because calico does not support eBPF for IPv6. Cilium supports IPv6 eBPF, but on the other hand does not support automatic BGP peering. So the following **does not** work: ``` kubeadm init --skip-phases=addon/kube-proxy --service-cidr 2a0a:e5c0:13:aaa::/108 --pod-network-cidr 2a0a:e5c0:13:bbb::/64 kubeadm init --service-cidr 2a0a:e5c0:13:aaa::/108 --pod-network-cidr 2a0a:e5c0:13:bbb::/64 ``` ## Worker nodes ``` kubeadm join [2a0a:e5c0:13:0:225:b3ff:fe20:38cc]:6443 --token bw3x98.chp31kcgcd4b5fpf --discovery-token-ca-cert-hash sha256:... ``` ## CNI/networking ``` kubectl apply -f calico.yaml ``` Warning: needs to be updated: ``` serviceaccount/calico-kube-controllers created Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget poddisruptionbudget.policy/calico-kube-controllers created ``` Checking pods: ``` [21:53] server47.place7:~/v3-calico# kubectl -n kube-system get pods NAME READY STATUS RESTARTS AGE calico-kube-controllers-6d8ccdbf46-4xzz9 0/1 Pending 0 60s calico-node-5gkp9 0/1 Init:0/3 0 60s calico-node-8lct9 0/1 Init:0/3 0 60s calico-node-jmjhn 0/1 Init:0/3 0 60s calico-node-krnzr 0/1 Init:ErrImagePull 0 60s coredns-558bd4d5db-4rvrf 0/1 Pending 0 3m40s coredns-558bd4d5db-g9lbx 0/1 Pending 0 3m40s etcd-server47 1/1 Running 0 3m56s kube-apiserver-server47 1/1 Running 0 3m55s kube-controller-manager-server47 1/1 Running 0 3m56s kube-scheduler-server47 1/1 Running 0 3m55s [21:54] server47.place7:~/v3-calico# ``` ## Getting calicoctl ``` kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml ``` And alias it: ``` alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl" ``` ## Configuring BGP routing ``` calcaticoctl create -f - < bgp....yaml ``` ## Setup a test deployment Do *NOT* use https://k8s.io/examples/application/deployment.yaml. It contains an outdated nginx container that has no IPv6 listener. You will get results such as ``` [19:03] server47.place7:~/ungleich-k8s/v3-calico# curl http://[2a0a:e5c0:13:bbb:176b:eaa6:6d47:1c41] curl: (7) Failed to connect to 2a0a:e5c0:13:bbb:176b:eaa6:6d47:1c41 port 80: Connection refused ``` if you use that deployment. Instead use something on the line of the included **nginx-test-deployment.yaml**: ``` kubectl apply -f nginx-test-deployment.yaml ``` ## Kubevirt Based on https://kubevirt.io/user-guide/operations/installation/: ``` export RELEASE=v0.42.1 # Deploy the KubeVirt operator kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml # Create the KubeVirt CR (instance deployment request) which triggers the actual installation kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml ``` This never happens: ``` # wait until all KubeVirt components are up $ kubectl -n kubevirt wait kv kubevirt --for condition=Available ``` ## Old/obsolete information ### Alpine / kubelet hack Due to some misconfiguration on alpine, **DURING** the **kubeadm init** we need to modify the **generated** /var/lib/kubelet/config.yaml to replace "cgroupDriver: systemd" with "cgroupDriver: cgroupfs". The same is necessary on the worker nodes, however that can be done anytime before you plan to schedule containers on them, after the **kubeadm join** request. **THIS is fixed if we use a kubeadm config file specifying the cgroupdriver**. ## Calico notes Requires ``` mount --make-shared /sys ``` ## Kubevirt nodes Requires ``` mount --make-shared / ``` ## Manual / post boot changes for place7-v2 cluster ``` mount --make-shared / mount --make-shared /sys sysctl net.ipv6.conf.eth0.accept_ra=2 modprobe br_netfilter sysctl -p /etc/sysctl.d/99-Z-sysctl-cdist.conf ``` ## Docker based ``` mount --make-shared /sys mount --make-shared /run/ ```