2021-06-06 16:12:15 +00:00
|
|
|
[[!meta title="Building an IPv6 only kubernetes cluster"]]
|
|
|
|
|
|
|
|
|
|
## Introduction
|
|
|
|
|
|
2021-06-09 17:11:03 +00:00
|
|
|
For a few weeks I am working on my pet project to create a production
|
2021-06-06 16:12:15 +00:00
|
|
|
ready kubernetes cluster that runs in an IPv6 only environment.
|
|
|
|
|
|
|
|
|
|
As the complexity and challenges for this project are rather
|
|
|
|
|
interesting, I decided to start documenting them in this blog post.
|
|
|
|
|
|
|
|
|
|
The
|
|
|
|
|
[ungleich-k8s](https://code.ungleich.ch/ungleich-public/ungleich-k8s)
|
|
|
|
|
contanins all snippets and latest code.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Objective
|
|
|
|
|
|
|
|
|
|
The kubernetes cluster should support the following work loads:
|
|
|
|
|
|
|
|
|
|
* Matrix Chat instances (Synapse+postgres+nginx+element)
|
|
|
|
|
* Virtual Machines (via kubevirt)
|
|
|
|
|
* Provide storage to internal and external consumers using Ceph
|
|
|
|
|
|
|
|
|
|
## Components
|
|
|
|
|
|
|
|
|
|
The following is a list of components that I am using so far. This
|
|
|
|
|
might change on the way, but I wanted to list already what I selected
|
|
|
|
|
and why.
|
|
|
|
|
|
|
|
|
|
### OS: Alpine Linux
|
|
|
|
|
|
|
|
|
|
The operating system of choice to run the k8s cluster is
|
|
|
|
|
[Alpine Linux](https://www.alpinelinux.org/) as it is small, stable
|
|
|
|
|
and supports both docker and cri-o.
|
|
|
|
|
|
|
|
|
|
### Container management: docker
|
|
|
|
|
|
|
|
|
|
Originally I started with [cri-o](https://cri-o.io/). However using
|
|
|
|
|
cri-o together with kubevirt and calico results in an overlayfs placed
|
|
|
|
|
on / of the host, which breaks the full host functionality (see below
|
|
|
|
|
for details).
|
|
|
|
|
|
|
|
|
|
Docker, while being deprecated, allows me to get kubevirt generally
|
|
|
|
|
speaking running.
|
|
|
|
|
|
|
|
|
|
### Networking: IPv6 only, calico
|
|
|
|
|
|
|
|
|
|
I wanted to go with [cilium](https://cilium.io/) first, because it
|
|
|
|
|
goes down the eBPF route from the get go. However cilium does not yet
|
|
|
|
|
contain native and automated BGP peering with the upstream
|
|
|
|
|
infrastructure, so managing nodes / ip network peering becomes a
|
|
|
|
|
tedious, manual and error prone task. Cilium is on the way to improve
|
|
|
|
|
this, but is not there yet.
|
|
|
|
|
|
|
|
|
|
[Calico](https://www.projectcalico.org/) on the other hand still
|
|
|
|
|
relies on ip(6)tables and kube-proxy for forwarding traffic, but has
|
|
|
|
|
for a long time proper BGP support. Calico also aims to add eBPF
|
|
|
|
|
support, however at the moment it does not support IPv6 yet (bummer!).
|
|
|
|
|
|
|
|
|
|
### Storage: rook
|
|
|
|
|
|
|
|
|
|
[Rook](https://rook.io/) seems to be the first choice if you search
|
|
|
|
|
who is doing what storage providers in the k8s world. It looks rather
|
|
|
|
|
proper, even though some knobs are not yet clear to me.
|
|
|
|
|
|
|
|
|
|
Rook, in my opinion, is a direct alternative of running cephadm, which
|
|
|
|
|
requires systemd running on your hosts. Which, given Alpine Linux,
|
|
|
|
|
will never be the case.
|
|
|
|
|
|
|
|
|
|
### Virtualisation
|
|
|
|
|
|
|
|
|
|
[Kubevirt](https://kubevirt.io/) seems to provide a good
|
|
|
|
|
interface. Mid term, kubevirt is projected to replace
|
|
|
|
|
[OpenNebula](https://opennebula.io/) at
|
|
|
|
|
[ungleich](https://ungleich.ch).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Challenges
|
|
|
|
|
|
|
|
|
|
### cri-o + calico + kubevirt = broken host
|
|
|
|
|
|
|
|
|
|
So this is a rather funky one. If you deploy cri-o and calico,
|
|
|
|
|
everything works. If you then deploy kubevirt, the **virt-handler**
|
|
|
|
|
pod fails to come up with the error message
|
|
|
|
|
|
|
|
|
|
Error: path "/var/run/kubevirt" is mounted on "/" but it is not a shared mount.
|
|
|
|
|
|
|
|
|
|
In the Internet there are two recommendations to fix this:
|
|
|
|
|
|
|
|
|
|
* Fix the systemd unit for docker: Obviously, using neither of them,
|
|
|
|
|
this is not applicable...
|
|
|
|
|
* Issue **mount --make-shared /**
|
|
|
|
|
|
|
|
|
|
The second command has a very strange side effect: Issueing that, the
|
|
|
|
|
contents of a calico pod are mounted as an overlayfs **on / of the
|
|
|
|
|
host**. This covers /proc and thus things like **ps**, **mount** and
|
|
|
|
|
co. fail and basically the whole system becomes unusable until reboot.
|
|
|
|
|
|
|
|
|
|
This is fully reproducible. I first suspected the tmpfs on / to be the
|
|
|
|
|
issue, used some disks instead of booting over network to check it and
|
|
|
|
|
even a regular ext4 on / causes the exact same problem.
|
|
|
|
|
|
|
|
|
|
### docker + calico + kubevirt = other shared mounts
|
|
|
|
|
|
|
|
|
|
Now, given that cri-o + calico + kubevirt does not lead to the
|
|
|
|
|
expected result, what does the same setup with docker look like? The
|
|
|
|
|
calico node pods with docker fail to come up, if /sys is not
|
|
|
|
|
shared mounted, the virt-handler pods fail if /run is not shared
|
|
|
|
|
mounted.
|
|
|
|
|
|
|
|
|
|
Two funky findings:
|
|
|
|
|
|
|
|
|
|
Issueing the following commands makes both work:
|
|
|
|
|
|
|
|
|
|
mount --make-shared /sys
|
|
|
|
|
mount --make-shared /run
|
|
|
|
|
|
|
|
|
|
The paths are totally different between docker and cri-o, even though
|
|
|
|
|
the mapped hostpaths in the pod description are the same. And why is
|
|
|
|
|
having /sys not being shared not a problem for calico in cri-o?
|
|
|
|
|
|
|
|
|
|
## Log
|
|
|
|
|
|
2021-06-07 18:43:54 +00:00
|
|
|
### Status 2021-06-07
|
|
|
|
|
|
|
|
|
|
Today I have updated the ceph cluster definition in rook to
|
|
|
|
|
|
|
|
|
|
* check hosts every 10 minutes instead of 60m for new disks
|
|
|
|
|
* use IPv6 instead of IPv6
|
|
|
|
|
|
2021-06-07 18:49:28 +00:00
|
|
|
The succesful ceph -s output:
|
|
|
|
|
|
|
|
|
|
[20:42] server47.place7:~/ungleich-k8s/rook# kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s
|
2021-06-07 18:43:54 +00:00
|
|
|
cluster:
|
|
|
|
|
id: 049110d9-9368-4750-b3d3-6ca9a80553d7
|
|
|
|
|
health: HEALTH_WARN
|
|
|
|
|
mons are allowing insecure global_id reclaim
|
|
|
|
|
|
|
|
|
|
services:
|
2021-06-07 18:49:28 +00:00
|
|
|
mon: 3 daemons, quorum a,b,d (age 75m)
|
|
|
|
|
mgr: a(active, since 74m), standbys: b
|
|
|
|
|
osd: 6 osds: 6 up (since 43m), 6 in (since 44m)
|
2021-06-07 18:43:54 +00:00
|
|
|
|
|
|
|
|
data:
|
|
|
|
|
pools: 2 pools, 33 pgs
|
|
|
|
|
objects: 6 objects, 34 B
|
|
|
|
|
usage: 37 MiB used, 45 GiB / 45 GiB avail
|
|
|
|
|
pgs: 33 active+clean
|
|
|
|
|
|
2021-06-07 18:49:28 +00:00
|
|
|
|
2021-06-07 18:43:54 +00:00
|
|
|
The result is a working ceph clusters with RBD support. I also applied
|
|
|
|
|
the cephfs manifest, however RWX volumes (readwritemany) are not yet
|
|
|
|
|
spinning up. It seems that test [helm charts](https://artifacthub.io/)
|
|
|
|
|
often require RWX instead of RWO (readwriteonce) access.
|
|
|
|
|
|
2021-06-07 18:46:26 +00:00
|
|
|
Also the ceph dashboard does not come up, even though it is
|
|
|
|
|
configured:
|
|
|
|
|
|
|
|
|
|
[20:44] server47.place7:~# kubectl -n rook-ceph get svc
|
|
|
|
|
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
|
|
|
csi-cephfsplugin-metrics ClusterIP 2a0a:e5c0:13:e2::760b <none> 8080/TCP,8081/TCP 82m
|
|
|
|
|
csi-rbdplugin-metrics ClusterIP 2a0a:e5c0:13:e2::482d <none> 8080/TCP,8081/TCP 82m
|
|
|
|
|
rook-ceph-mgr ClusterIP 2a0a:e5c0:13:e2::6ab9 <none> 9283/TCP 77m
|
|
|
|
|
rook-ceph-mgr-dashboard ClusterIP 2a0a:e5c0:13:e2::5a14 <none> 7000/TCP 77m
|
|
|
|
|
rook-ceph-mon-a ClusterIP 2a0a:e5c0:13:e2::c39e <none> 6789/TCP,3300/TCP 83m
|
|
|
|
|
rook-ceph-mon-b ClusterIP 2a0a:e5c0:13:e2::732a <none> 6789/TCP,3300/TCP 81m
|
|
|
|
|
rook-ceph-mon-d ClusterIP 2a0a:e5c0:13:e2::c658 <none> 6789/TCP,3300/TCP 76m
|
|
|
|
|
[20:44] server47.place7:~# curl http://[2a0a:e5c0:13:e2::5a14]:7000
|
|
|
|
|
curl: (7) Failed to connect to 2a0a:e5c0:13:e2::5a14 port 7000: Connection refused
|
|
|
|
|
[20:45] server47.place7:~#
|
|
|
|
|
|
|
|
|
|
The ceph mgr is perfectly reachable though:
|
|
|
|
|
|
|
|
|
|
[20:45] server47.place7:~# curl -s http://[2a0a:e5c0:13:e2::6ab9]:9283/metrics | head
|
|
|
|
|
|
|
|
|
|
# HELP ceph_health_status Cluster health status
|
|
|
|
|
# TYPE ceph_health_status untyped
|
|
|
|
|
ceph_health_status 1.0
|
|
|
|
|
# HELP ceph_mon_quorum_status Monitors in quorum
|
|
|
|
|
# TYPE ceph_mon_quorum_status gauge
|
|
|
|
|
ceph_mon_quorum_status{ceph_daemon="mon.a"} 1.0
|
|
|
|
|
ceph_mon_quorum_status{ceph_daemon="mon.b"} 1.0
|
|
|
|
|
ceph_mon_quorum_status{ceph_daemon="mon.d"} 1.0
|
|
|
|
|
# HELP ceph_fs_metadata FS Metadata
|
|
|
|
|
|
|
|
|
|
|
2021-06-06 16:12:15 +00:00
|
|
|
### Status 2021-06-06
|
|
|
|
|
|
|
|
|
|
Today is the first day of publishing the findings and this blog
|
|
|
|
|
article will lack quite some information. If you are curious and want
|
|
|
|
|
to know more that is not yet published, you can find me on Matrix
|
|
|
|
|
in the **#hacking:ungleich.ch** room.
|
|
|
|
|
|
2021-06-07 18:43:54 +00:00
|
|
|
#### What works so far
|
2021-06-06 16:12:15 +00:00
|
|
|
|
|
|
|
|
* Spawing pods IPv6 only
|
|
|
|
|
* Spawing IPv6 only services works
|
|
|
|
|
* BGP Peering and ECMP routes with the upstream infrastructure works
|
|
|
|
|
|
|
|
|
|
Here's an output of the upstream bird process for the routes from k8s:
|
|
|
|
|
|
|
|
|
|
bird> show route
|
|
|
|
|
Table master6:
|
|
|
|
|
2a0a:e5c0:13:e2::/108 unicast [place7-server1 23:45:21.589] * (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 on eth0
|
|
|
|
|
unicast [place7-server3 2021-06-05] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:224:81ff:fee0:db7a on eth0
|
|
|
|
|
unicast [place7-server4 2021-06-05] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 on eth0
|
|
|
|
|
unicast [place7-server2 2021-06-05] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc on eth0
|
|
|
|
|
2a0a:e5c0:13:e1:176b:eaa6:6d47:1c40/122 unicast [place7-server1 23:45:21.589] * (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 on eth0
|
|
|
|
|
unicast [place7-server4 23:45:21.591] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 on eth0
|
|
|
|
|
unicast [place7-server3 23:45:21.591] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:224:81ff:fee0:db7a on eth0
|
|
|
|
|
unicast [place7-server2 23:45:21.589] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc on eth0
|
|
|
|
|
2a0a:e5c0:13:e1:e0d1:d390:343e:8480/122 unicast [place7-server1 23:45:21.589] * (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 on eth0
|
|
|
|
|
unicast [place7-server3 2021-06-05] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:224:81ff:fee0:db7a on eth0
|
|
|
|
|
unicast [place7-server4 2021-06-05] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 on eth0
|
|
|
|
|
unicast [place7-server2 2021-06-05] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc on eth0
|
|
|
|
|
2a0a:e5c0:13::/48 unreachable [v6 2021-05-16] * (200)
|
|
|
|
|
2a0a:e5c0:13:e1:9b19:7142:bebb:4d80/122 unicast [place7-server1 23:45:21.589] * (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 on eth0
|
|
|
|
|
unicast [place7-server3 2021-06-05] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:224:81ff:fee0:db7a on eth0
|
|
|
|
|
unicast [place7-server4 2021-06-05] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 on eth0
|
|
|
|
|
unicast [place7-server2 2021-06-05] (100) [AS65534i]
|
|
|
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc on eth0
|
|
|
|
|
bird>
|
|
|
|
|
|
|
|
|
|
|
2021-06-07 18:43:54 +00:00
|
|
|
#### What doesn't work
|
2021-06-06 16:12:15 +00:00
|
|
|
|
|
|
|
|
* Rook does not format/spinup all disks
|
|
|
|
|
* Deleting all rook components fails (**kubectl delete -f cluster.yaml
|
|
|
|
|
hangs** forever)
|
|
|
|
|
* Spawning VMs fails with **error: unable to recognize "vmi.yaml": no matches for kind "VirtualMachineInstance" in version "kubevirt.io/v1"**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
[[!tag kubernetes ipv6]]
|
