diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr new file mode 100644 index 0000000..d0ee04c --- /dev/null +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -0,0 +1,91 @@ +title: [WIP] Migrating Ceph Nautilus into Kubernetes + Rook +--- +pub_date: 2022-08-27 +--- +author: ungleich storage team +--- +twitter_handle: ungleich +--- +_hidden: no +--- +_discoverable: yes +--- +abstract: +How we move our Ceph clusters into kubernetes +--- +body: + +## Introduction + +At ungleich we are running multiple Ceph clusters. Some of them are +running Ceph Nautilus (14.x) based on +[Devuan](https://www.devuan.org/). Our newer Ceph Pacific (16.x) +clusters are running based on [Rook](https://rook.io/) on +[Kubernetes](https://kubernetes.io/) on top of +[Alpine Linux](https://alpinelinux.org/). + +In this blog article we will describe how to migrate +Ceph/Native/Devuan to Ceph/k8s+rook/Alpine Linux. + +## Work in Progress [WIP] + +This blog article is work in progress. The migration planning has +started, however the migration has not been finished yet. This article +will feature the different paths we take for the migration. + +## The Plan + +To continue operating the cluster during the migration, the following +steps are planned: + +* Setup a k8s cluster that can potentially communicate with the + existing ceph cluster +* Using the [disaster + recovery](https://rook.io/docs/rook/v1.9/Troubleshooting/disaster-recovery/) + guidelines from rook to modify the rook configuration to use the + previous fsid. +* Spin up ceph monitors and ceph managers in rook +* Retire existing monitors +* Shutdown a ceph OSD node, remove it's OS disk, boot it with Alpine + Linux +* Join the node into the k8s cluster +* Have rook pickup the existing disks and start the osds +* Repeat if successful +* Migrate to ceph pacific + +## Original cluster + +The target ceph cluster we want to migrate lives in the 2a0a:e5c0::/64 +network. Ceph is using: + +``` +public network = 2a0a:e5c0:0:0::/64 +cluster network = 2a0a:e5c0:0:0::/64 +``` + +## Kubernetes cluster networking inside the ceph network + +To be able to communicate with the existing OSDs, we will be using +sub networks of 2a0a:e5c0::/64 for kubernetes. As these networks +are part of the link assigned network 2a0a:e5c0::/64, we will use BGP +routing on the existing ceph nodes to create more specific routes into +the kubernetes cluster. + +As we plan to use either [cilium](https://cilium.io/) or +[calico](https://www.tigera.io/project-calico/) as the CNI, we can +configure kubernetes to directly BGP peer with the existing Ceph +nodes. + + +## Changelog + +### 2022-08-27 + +* The initial release of this blog article + + +## Follow up or questions + +You can join the discussion in the matrix room +#kubernetes:ungleich.ch about this migration. If don't have a matrix +account you can join using our chat on https://chat.with.ungleich.ch.