From e4f4fb7194bd6c9e58a9e9f8c52843d108c980bd Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 3 Jul 2022 13:30:44 +0200 Subject: [PATCH 01/22] Update Kubernetes hosting --- .../contents.lr | 0 .../low-power-kubernetes-hosting/contents.lr | 239 ++++++++++++++++++ 2 files changed, 239 insertions(+) rename content/u/products/{kubernetes-cluster => kubernetes-hosting}/contents.lr (100%) create mode 100644 content/u/products/low-power-kubernetes-hosting/contents.lr diff --git a/content/u/products/kubernetes-cluster/contents.lr b/content/u/products/kubernetes-hosting/contents.lr similarity index 100% rename from content/u/products/kubernetes-cluster/contents.lr rename to content/u/products/kubernetes-hosting/contents.lr diff --git a/content/u/products/low-power-kubernetes-hosting/contents.lr b/content/u/products/low-power-kubernetes-hosting/contents.lr new file mode 100644 index 0000000..62d69dc --- /dev/null +++ b/content/u/products/low-power-kubernetes-hosting/contents.lr @@ -0,0 +1,239 @@ +_discoverable: yes +--- +_hidden: no +--- +title: Low Power Kubernetes Clusters +--- +subtitle: Kubernetes the green way +--- +headline1: Environment Friendly +--- +headline2: Scalable & Managed +--- +headline3: Kubernetes Clusters +--- +header_background_color: #316CE6 +--- +header_text_color: text-light +--- +nav_classes: navbar-dark +--- +image:/u/image/cards/kubernetes.jpg +--- +feature1_title: Environment Friendly +--- +feature1_text: +You want to run Kubernetes in a sustainable and affordable way? Then +this Kubernetes hosting is for you: Built specifically on low power +components, your Kubernetes cluster is not only friendly to your +wallet, but even more to the environment. +--- +feature2_title: Scalable +--- +feature2_text: + +Your Kubernetes cluster can start from 3 nodes and grow to any size. Every +node that you add to your cluster contributes additional CPU, RAM and +persistent storage. +--- +feature3_title: Dedicated +--- +feature3_text: +Your Kubernetes cluster is running on dedicated hardware. No +interruption, no rescheduling, no peer pressure. All resources are +100% usable by you. +--- +feature4_title: Very Environment friendly +--- +feature4_text: +Each Kubernetes node has been tested to consume less than 20W at +maximum performance. While our data center is using 100% renewable +energy, we try to even make an extra effort to reduce impact on the +environment. +--- +feature6_title: 100% IPv6 +--- +feature6_text: +The Kubernetes cluster is ready for the IPv6 world: all nodes are +running on an IPv6-only network. For IPv4 reachability we offer you +dedicated mapped IPv4 addresses. + +--- +content1_text: + +## Hosted Kubernetes Clusters by ungleich + +While you might be aware of other managed Kubernetes clusters, here at +ungleich we have quite a different offering. Let's start with the +obvious ones that you are unlikely to experience anywhere else: + +### Environment friendly, for real + +Other providers are dealing with Carbon Certificates, offsetting their +energy and claiming to be green. None of that makes sense to us, as +the emissions are produced nonetheless, fossil fuels are used or +nuclear energy contributes to running your Kubernetes cluster +elsewhere. + +Here at ungleich we start from the beginning and continue to improve +our energy usage every day. Not only are we using a 100% renewable +energy (locally produced!), but we also research specific hardware to +optimise the energy usage required by Kubernetes clusters. + +## Standard building blocks + +Other providers offer you a complex system of building blocks: this +CPU, that memory, different storage classes, bandwidth consumption and +so on and so on. You easily lose the track of what you wanted to do in +the first place: to run your workload. + +Here at ungleich we offer you exactly 1 building block to build your +kubernetes clusters: The ungleich k8s node. + +## The ungleich k8s node + +The ungleich k8s node is an AMD based, 12 thread computer with 64 GB +of main memory and two SSDs. The first SSDs is used for storing the +operating system and the second SSD is used to store your data. + +The second SSD has a capacity of 1TB. + +## The ungleich kubernetes cluster + +Each Kubernetes cluster consists of 3 or more k8s nodes. While you can +grow it with adding individual nodes, we recommend to grow it in +numbers of 3. The reason for this is the underlying storage technology +[Ceph](https://ceph.com/) that saves each of your data blocks 3 times. + +## Failure resistent + +As your workload is running on any of the k8s nodes, maintenance work +and failovers can be executed without interrupting the services +(assuming your workload is deployed accordingly). + +## Storage support included + +Using a built-in dedicated Ceph cluster we provide you with persistent +storage that can be accessed using standard Kubernetes PVCs. + +As every k8s node contains a dedicated SSD, by adding new nodes, your +storage cluster automatically grows as well. + +As the Ceph cluster stores every object 3 times, the 1 TB SSD of +each node contributes about 300GiB usable storage. + + +## Pricing + +Our pricing is very simple: Per K8S node we charge 300 CHF/month. This +includes rental of the hardware, network traffic, electricity and +maintenance of the kubernetes cluster. + +This might not be the cheapest kubernetes you can build, but it is +probably the world's most energy aware and most modern Kubernetes +cluster out there. + +--- +offer1_title: 3 Node k8s cluster 900 CHF/month +--- +offer1_text: + +* [One time setup fee 150 CHF](https://datacenterlight.ch/product/k8s-setup-3) +* 3 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD +* Total 36 threads, 192 GB RAM, ca. 1 TB persistent storage +* Suitable for a development use + +--- +offer1_link: https://datacenterlight.ch/product/k8s-hosting-3 +--- +offer2_title: 6 Node k8s cluster 1800 CHF/month +--- +offer2_text: + +* [One time setup fee 300 CHF](https://datacenterlight.ch/product/k8s-setup-6/) +* 6 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD +* Total 72 threads, 384 GB RAM, ca. 2 TB persistent storage +* Suitable as small production cluster + +--- +offer2_link: https://datacenterlight.ch/product/k8s-hosting-6 +--- +offer3_title: 9 Node k8s cluster 2700 CHF/month +--- +offer3_text: + +* [One time setup fee 450 CHF](https://datacenterlight.ch/product/k8s-setup-9/) +* 9 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD +* Total: 108 threads, 576GB RAM, ca. 3 TB persistent storage +* Suitable as a production cluster + +--- +offer3_link: https://datacenterlight.ch/product/k8s-hosting-9 +--- +offer4_title: 12 Node k8s cluster 3600 CHF/month +--- +offer4_text: + +* [One time setup fee 600 CHF](https://datacenterlight.ch/product/k8s-setup-12/) +* 12 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD +* Total: 144 threads, 768 GB RAM, ca. 4 TB persistent storage +* Suitable as a production cluster + +--- +offer4_link: https://datacenterlight.ch/product/k8s-hosting-12 +--- +offer5_title: 15 Node k8s cluster 4500 CHF/month +--- +offer5_text: + +* [One time setup fee 750 CHF](https://datacenterlight.ch/product/k8s-setup-15/) +* 3 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD +* Total: 180 threads, 960 GB RAM, ca. 5 TB persistent storage +* Suitable as a large production cluster + +--- +offer5_link: https://datacenterlight.ch/product/k8s-hosting-15 +--- +offer6_title: 18 Node k8s cluster 5400 CHF/month +--- +offer6_text: + +* [One time setup fee 900 CHF](https://datacenterlight.ch/product/k8s-setup-18/) +* 3 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD +* Total: 216 threads, 1152 GB RAM, ca. 6 TB persistent storage +* Suitable as a large production cluster + +--- +offer6_link: https://datacenterlight.ch/product/k8s-hosting-18 + +--- +content2_text: + +## Tech Details + +### The ungleich k8s node + +* CPU: AMD Ryzen 5 5600G +* RAM: 64 GB DDR4-3200 +* OS SSD: M2, 128 GB or more +* Data SSD: SATA, 1 TB, usually ca. 500 MB read/write (work load dependent) +* Connectivity: 1 Gbit/s +* OS: Alpine Linux +* Networking: IPv6 only + +### The ungleich k8s cluster + +The Kubernetes control plane access is available via IPv6 and +IPv4. All pods are running in a dedicated /64 IPv6 network. +Service IPs are IPv6 only and are located in a second /64 IPv6 network. + +The cluster does not support type=LoadBalancer, because it is not +necessary in IPv6 only environments. ClusterIPs are already by default +world reachable. + +IPv4 reachablity can be configured in two ways: + +* Via our IPv4-to-IPv6 proxy (http, https based) +* Via a dedicated IPv4 address + +The current stack uses Calico as the CNI. From 7b467981042f520852084414525e7742361d1bc9 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 3 Jul 2022 13:38:30 +0200 Subject: [PATCH 02/22] Rename again --- .../contents.lr | 2 +- .../contents.lr | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename content/u/products/{kubernetes-hosting => kubernetes-hosting-high-performance}/contents.lr (99%) rename content/u/products/{low-power-kubernetes-hosting => kubernetes-hosting-low-power}/contents.lr (100%) diff --git a/content/u/products/kubernetes-hosting/contents.lr b/content/u/products/kubernetes-hosting-high-performance/contents.lr similarity index 99% rename from content/u/products/kubernetes-hosting/contents.lr rename to content/u/products/kubernetes-hosting-high-performance/contents.lr index cee4b4c..acba821 100644 --- a/content/u/products/kubernetes-hosting/contents.lr +++ b/content/u/products/kubernetes-hosting-high-performance/contents.lr @@ -2,7 +2,7 @@ _discoverable: yes --- _hidden: no --- -title: Managed Kubernetes Clusters +title: High Perforance Kubernetes Hosting --- subtitle: Scalable and climate friendly --- diff --git a/content/u/products/low-power-kubernetes-hosting/contents.lr b/content/u/products/kubernetes-hosting-low-power/contents.lr similarity index 100% rename from content/u/products/low-power-kubernetes-hosting/contents.lr rename to content/u/products/kubernetes-hosting-low-power/contents.lr From 5a47d64eb9c477ced31c5a7d871d72339ca3cc3f Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 3 Jul 2022 13:40:25 +0200 Subject: [PATCH 03/22] change heading --- .../u/products/kubernetes-hosting-high-performance/contents.lr | 2 +- content/u/products/kubernetes-hosting-low-power/contents.lr | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/content/u/products/kubernetes-hosting-high-performance/contents.lr b/content/u/products/kubernetes-hosting-high-performance/contents.lr index acba821..abffed4 100644 --- a/content/u/products/kubernetes-hosting-high-performance/contents.lr +++ b/content/u/products/kubernetes-hosting-high-performance/contents.lr @@ -2,7 +2,7 @@ _discoverable: yes --- _hidden: no --- -title: High Perforance Kubernetes Hosting +title: Kubernetes Hosting - High Performance --- subtitle: Scalable and climate friendly --- diff --git a/content/u/products/kubernetes-hosting-low-power/contents.lr b/content/u/products/kubernetes-hosting-low-power/contents.lr index 62d69dc..5c08a31 100644 --- a/content/u/products/kubernetes-hosting-low-power/contents.lr +++ b/content/u/products/kubernetes-hosting-low-power/contents.lr @@ -2,7 +2,7 @@ _discoverable: yes --- _hidden: no --- -title: Low Power Kubernetes Clusters +title: Kubernetes Cluster - Low Power --- subtitle: Kubernetes the green way --- From 7ec5626339cb90bd10afcb181675898fe5839aff Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 3 Jul 2022 14:08:26 +0200 Subject: [PATCH 04/22] ++offer --- .../contents.lr | 4 +- .../kubernetes-hosting-low-power/contents.lr | 182 +++++++----------- 2 files changed, 68 insertions(+), 118 deletions(-) diff --git a/content/u/products/kubernetes-hosting-high-performance/contents.lr b/content/u/products/kubernetes-hosting-high-performance/contents.lr index abffed4..feeb96d 100644 --- a/content/u/products/kubernetes-hosting-high-performance/contents.lr +++ b/content/u/products/kubernetes-hosting-high-performance/contents.lr @@ -4,7 +4,7 @@ _hidden: no --- title: Kubernetes Hosting - High Performance --- -subtitle: Scalable and climate friendly +subtitle: Fast and climate friendly --- headline1: Environment Friendly --- @@ -147,7 +147,7 @@ offer1_text: * [One time setup fee 150 CHF](https://datacenterlight.ch/product/k8s-setup-3) * 3 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD * Total 36 threads, 192 GB RAM, ca. 1 TB persistent storage -* Suitable for a development use +* Suitable ONLY for development use (control plane acts as workers) --- offer1_link: https://datacenterlight.ch/product/k8s-hosting-3 diff --git a/content/u/products/kubernetes-hosting-low-power/contents.lr b/content/u/products/kubernetes-hosting-low-power/contents.lr index 5c08a31..64aa296 100644 --- a/content/u/products/kubernetes-hosting-low-power/contents.lr +++ b/content/u/products/kubernetes-hosting-low-power/contents.lr @@ -4,7 +4,7 @@ _hidden: no --- title: Kubernetes Cluster - Low Power --- -subtitle: Kubernetes the green way +subtitle: Sustainable Kubernetes Hosting --- headline1: Environment Friendly --- @@ -27,47 +27,47 @@ You want to run Kubernetes in a sustainable and affordable way? Then this Kubernetes hosting is for you: Built specifically on low power components, your Kubernetes cluster is not only friendly to your wallet, but even more to the environment. ---- -feature2_title: Scalable ---- -feature2_text: -Your Kubernetes cluster can start from 3 nodes and grow to any size. Every -node that you add to your cluster contributes additional CPU, RAM and -persistent storage. ---- -feature3_title: Dedicated ---- -feature3_text: -Your Kubernetes cluster is running on dedicated hardware. No -interruption, no rescheduling, no peer pressure. All resources are -100% usable by you. ---- -feature4_title: Very Environment friendly ---- -feature4_text: Each Kubernetes node has been tested to consume less than 20W at maximum performance. While our data center is using 100% renewable energy, we try to even make an extra effort to reduce impact on the environment. + --- -feature6_title: 100% IPv6 +feature2_title: Affordable and Scalable --- -feature6_text: -The Kubernetes cluster is ready for the IPv6 world: all nodes are -running on an IPv6-only network. For IPv4 reachability we offer you -dedicated mapped IPv4 addresses. +feature2_text: + +The low power Kubernetes hosting not only is very environment +friendly, but also very friendly to the budget. By using significantly +less resources, you contribute to saving the environment as well as +spending money on resources you don't need. + +When starting small, you always have the possibility to grow your +cluster in the future, reacting to the actual needs. +--- +feature3_title: Dedicated & 100% IPv6 +--- +feature3_text: +Your Kubernetes cluster is running on dedicated hardware. No +interruption, no rescheduling, no peer pressure. All resources are +completely usable by you. + +All our Kubernetes cluster are fully ready for the IPv6 world: +all nodes are running on an IPv6-only network. For IPv4 reachability +we offer you dedicated mapped IPv4 addresses. --- content1_text: -## Hosted Kubernetes Clusters by ungleich +## The ungleich Low Power Kubernetes Hosting offer -While you might be aware of other managed Kubernetes clusters, here at -ungleich we have quite a different offering. Let's start with the -obvious ones that you are unlikely to experience anywhere else: +Our Kubernetes Hosting might be the world's most environment friendly +Kubernetes hosting out there. Not only are all components carefully +chosen to use as little energy as possible, but additionally the +electricity is coming from 100% renewebale energy sources. -### Environment friendly, for real +## Environment friendly, for real Other providers are dealing with Carbon Certificates, offsetting their energy and claiming to be green. None of that makes sense to us, as @@ -80,25 +80,7 @@ our energy usage every day. Not only are we using a 100% renewable energy (locally produced!), but we also research specific hardware to optimise the energy usage required by Kubernetes clusters. -## Standard building blocks - -Other providers offer you a complex system of building blocks: this -CPU, that memory, different storage classes, bandwidth consumption and -so on and so on. You easily lose the track of what you wanted to do in -the first place: to run your workload. - -Here at ungleich we offer you exactly 1 building block to build your -kubernetes clusters: The ungleich k8s node. - -## The ungleich k8s node - -The ungleich k8s node is an AMD based, 12 thread computer with 64 GB -of main memory and two SSDs. The first SSDs is used for storing the -operating system and the second SSD is used to store your data. - -The second SSD has a capacity of 1TB. - -## The ungleich kubernetes cluster +## Each Kubernetes cluster consists of 3 or more k8s nodes. While you can grow it with adding individual nodes, we recommend to grow it in @@ -123,112 +105,80 @@ As the Ceph cluster stores every object 3 times, the 1 TB SSD of each node contributes about 300GiB usable storage. -## Pricing +## Simple Pricing -Our pricing is very simple: Per K8S node we charge 300 CHF/month. This -includes rental of the hardware, network traffic, electricity and -maintenance of the kubernetes cluster. +Our pricing is very simple: Each pack of 3 node nodes is 225 +CHF/month. This includes rental of the hardware, network traffic, +electricity and maintenance of the kubernetes cluster. -This might not be the cheapest kubernetes you can build, but it is -probably the world's most energy aware and most modern Kubernetes -cluster out there. +You don't have to worry about additional storage, network fees or +other hidden costs. + +## Need more power? + +When you start with our low power Kubernetes hosting and at some point +in the future you realise that your applications need more resources, +you can even combine the low power Kubernetes hosting with the +[High Performance Kubernetes +Hosting](../kubernetes-hosting-high-performance). --- -offer1_title: 3 Node k8s cluster 900 CHF/month +offer1_title: 3 Node Low Power Kubernetes cluster 225 CHF/month --- offer1_text: * [One time setup fee 150 CHF](https://datacenterlight.ch/product/k8s-setup-3) -* 3 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD -* Total 36 threads, 192 GB RAM, ca. 1 TB persistent storage -* Suitable for a development use - +* 3 nodes with each: 4 Threads, 8 GB RAM, 1 TB Storage SSD +* Total 12 threads, 24 GB RAM, ca. 1 TB persistent storage +* Suitable ONLY for development use (control plane acts as workers) --- -offer1_link: https://datacenterlight.ch/product/k8s-hosting-3 +offer1_link: https://datacenterlight.ch/product/low-power-k8s-hosting-3 --- -offer2_title: 6 Node k8s cluster 1800 CHF/month +offer2_title: 6 Node Low Power Kubernetes cluster 450 CHF/month --- offer2_text: * [One time setup fee 300 CHF](https://datacenterlight.ch/product/k8s-setup-6/) -* 6 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD -* Total 72 threads, 384 GB RAM, ca. 2 TB persistent storage -* Suitable as small production cluster +* 6 nodes with each: 4 Threads, 8 GB RAM, 1 TB Storage SSD +* Total 24 threads, 48 GB RAM, ca. 2 TB persistent storage +* Suitable as small, low power production cluster --- -offer2_link: https://datacenterlight.ch/product/k8s-hosting-6 +offer2_link: https://datacenterlight.ch/product/low-power-k8s-hosting-6 --- -offer3_title: 9 Node k8s cluster 2700 CHF/month +offer3_title: 9 Node Low Power Kubernetes cluster 675 CHF/month --- offer3_text: * [One time setup fee 450 CHF](https://datacenterlight.ch/product/k8s-setup-9/) -* 9 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD -* Total: 108 threads, 576GB RAM, ca. 3 TB persistent storage -* Suitable as a production cluster - +* 9 nodes with each: 4 Threads, 8 GB RAM, 1 TB Storage SSD +* Total 36 threads, 72 GB RAM, ca. 3 TB persistent storage +* Suitable as a low power production cluster --- -offer3_link: https://datacenterlight.ch/product/k8s-hosting-9 ---- -offer4_title: 12 Node k8s cluster 3600 CHF/month ---- -offer4_text: - -* [One time setup fee 600 CHF](https://datacenterlight.ch/product/k8s-setup-12/) -* 12 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD -* Total: 144 threads, 768 GB RAM, ca. 4 TB persistent storage -* Suitable as a production cluster - ---- -offer4_link: https://datacenterlight.ch/product/k8s-hosting-12 ---- -offer5_title: 15 Node k8s cluster 4500 CHF/month ---- -offer5_text: - -* [One time setup fee 750 CHF](https://datacenterlight.ch/product/k8s-setup-15/) -* 3 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD -* Total: 180 threads, 960 GB RAM, ca. 5 TB persistent storage -* Suitable as a large production cluster - ---- -offer5_link: https://datacenterlight.ch/product/k8s-hosting-15 ---- -offer6_title: 18 Node k8s cluster 5400 CHF/month ---- -offer6_text: - -* [One time setup fee 900 CHF](https://datacenterlight.ch/product/k8s-setup-18/) -* 3 nodes with each: 12 Threads, 64GB RAM, 1 TB Storage SSD -* Total: 216 threads, 1152 GB RAM, ca. 6 TB persistent storage -* Suitable as a large production cluster - ---- -offer6_link: https://datacenterlight.ch/product/k8s-hosting-18 - +offer3_link: https://datacenterlight.ch/product/low-power-k8s-hosting-9 --- content2_text: ## Tech Details -### The ungleich k8s node +### The Low Power kubernetes node -* CPU: AMD Ryzen 5 5600G -* RAM: 64 GB DDR4-3200 -* OS SSD: M2, 128 GB or more -* Data SSD: SATA, 1 TB, usually ca. 500 MB read/write (work load dependent) +* CPU: [Intel Pentium CPU J4205](https://www.intel.com/content/www/us/en/products/sku/95591/intel-pentium-processor-j4205-2m-cache-up-to-2-60-ghz/specifications.html) +* RAM: 8 GB DDR3 +* OS SSD: 128 GB +* Data SSD: USB-C, 1 TB * Connectivity: 1 Gbit/s * OS: Alpine Linux * Networking: IPv6 only -### The ungleich k8s cluster +### The ungleich Kubernetes cluster The Kubernetes control plane access is available via IPv6 and IPv4. All pods are running in a dedicated /64 IPv6 network. Service IPs are IPv6 only and are located in a second /64 IPv6 network. The cluster does not support type=LoadBalancer, because it is not -necessary in IPv6 only environments. ClusterIPs are already by default +necessary in IPv6 only environments. ClusterIPs are already world reachable. IPv4 reachablity can be configured in two ways: From ee704c9500c98f5cb62bf66cfbe7af1e0a7b4ca6 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 3 Jul 2022 14:12:44 +0200 Subject: [PATCH 05/22] Link Kubernetes offers --- .../products/kubernetes-hosting-high-performance/contents.lr | 4 ++++ content/u/products/kubernetes-hosting-low-power/contents.lr | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/content/u/products/kubernetes-hosting-high-performance/contents.lr b/content/u/products/kubernetes-hosting-high-performance/contents.lr index feeb96d..1ca6bd9 100644 --- a/content/u/products/kubernetes-hosting-high-performance/contents.lr +++ b/content/u/products/kubernetes-hosting-high-performance/contents.lr @@ -243,3 +243,7 @@ IPv4 reachablity can be configured in two ways: * Via a dedicated IPv4 address The current stack uses Calico as the CNI. + +## Related products + +* Checkout the [Low Power Kubernetes Hosting](../kubernetes-hosting-low-power) diff --git a/content/u/products/kubernetes-hosting-low-power/contents.lr b/content/u/products/kubernetes-hosting-low-power/contents.lr index 64aa296..bada508 100644 --- a/content/u/products/kubernetes-hosting-low-power/contents.lr +++ b/content/u/products/kubernetes-hosting-low-power/contents.lr @@ -187,3 +187,7 @@ IPv4 reachablity can be configured in two ways: * Via a dedicated IPv4 address The current stack uses Calico as the CNI. + +## Related products + +* Checkout the [High Performance Kubernetes Hosting](../kubernetes-hosting-high-performance) From 750d94d9f0ab8ce4174bcac114f88ab6d07dd559 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 10 Jul 2022 21:27:23 +0200 Subject: [PATCH 06/22] [blog] phase in conduit --- .../contents.lr | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr diff --git a/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr b/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr new file mode 100644 index 0000000..5f16736 --- /dev/null +++ b/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr @@ -0,0 +1,58 @@ +title: [matrix] Conduit beta at ungleich +--- +pub_date: 2022-07-10 +--- +author: ungleich matrix team +--- +twitter_handle: ungleich +--- +_hidden: no +--- +_discoverable: yes +--- +abstract: +Let's give Conduit a try! IPv6 only, of course. +--- +body: + +## News from the Matrix Home Server department + +Here at ungleich we always have a look around, which Matrix home +server to use. Obviously, you all know about +[synapse](https://github.com/matrix-org/synapse), the reference +server. +Some of you might also know about +[dendrite](https://github.com/matrix-org/dendrite), an experimental, +maybe at some point in the future replacement of synapse. + +But did you know about [conduit](https://conduit.rs/), a tiny Matrix +homeserver written in Rust? Well, now you do. + +## Next generation Matrix Home Server? + +At ungleich we like to see what is coming next and conduit certainly +has the potential to become an interesting, production grade +homeserver. At the moment there are some +[issues open for the 1.0 +milestone](https://gitlab.com/famedly/conduit/-/milestones/3#tab-issues), +however conduit is actually already quite usable. + +How stable you ask? Well, ... + +## Phasing in public beta server at ungleich + +To give you an preview, we have setup an **IPv6 only** instance of +conduit at [conduit.ungleich.ch](https://conduit.ungleich.ch). + +This server is open for public use, anyone (well, anyone with +IPv6...) can register. However note: **this is only a test +instance**. We might at any point in the future delete, replace or +reset it. It is not backed and not a production service. It's main +intent is to allow you to easily test out Conduit. + +If you are looking for a production grade homeserver for yourself, +have a look at our [hosted Matrix +offerings](https://ungleich.ch/u/products/hosted-matrix-chat/). + +That said - give it a shot, we think Conduit is a promising +homeserver, especially given its focus on low resource usage. From 66b09aeb156c23a38071046ed2b0d7e601ed0971 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 10 Jul 2022 21:36:29 +0200 Subject: [PATCH 07/22] ++IPv6 only notes --- .../contents.lr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr b/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr index 5f16736..9ed474e 100644 --- a/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr +++ b/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr @@ -56,3 +56,18 @@ offerings](https://ungleich.ch/u/products/hosted-matrix-chat/). That said - give it a shot, we think Conduit is a promising homeserver, especially given its focus on low resource usage. + +## ... why IPv6 only? + +There are several reason for it: + +* IPv6 only is much easier to setup than IPv6+IPv4 +* This is a public beta service for which we try to keep the effort + low +* It's good test for us to see if conduit works in our IPv6 only + networks +* Global IPv6 traffic is nearing 50% and [many countries already + exceed 50%](https://stats.labs.apnic.net/ipv6) + +If you cannot access the conduit server because you don't have IPv6, +you can always get IPv6 via [the IPv6VPN](/u/products/ipv6-vpn/). From 782fb8b7dfa47f813073f014b5382ed02ddf8b0d Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 10 Jul 2022 21:37:27 +0200 Subject: [PATCH 08/22] ++contact --- .../contents.lr | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr b/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr index 9ed474e..5df5fff 100644 --- a/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr +++ b/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr @@ -71,3 +71,9 @@ There are several reason for it: If you cannot access the conduit server because you don't have IPv6, you can always get IPv6 via [the IPv6VPN](/u/products/ipv6-vpn/). + +## Follow up or questions + +You can join the discussion either on the conduit server or you can +also join our chat on https://chat.with.ungleich.ch (IPv6+IPv4 Synapse +based Matrix server). From 082ab313b52e4e4b70af418b5722cf7baf1e88cd Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 10 Jul 2022 21:38:08 +0200 Subject: [PATCH 09/22] ++conduit room --- .../contents.lr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr b/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr index 5df5fff..90cfe07 100644 --- a/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr +++ b/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr @@ -77,3 +77,6 @@ you can always get IPv6 via [the IPv6VPN](/u/products/ipv6-vpn/). You can join the discussion either on the conduit server or you can also join our chat on https://chat.with.ungleich.ch (IPv6+IPv4 Synapse based Matrix server). + +If you have anything to share about conduit, feel free to join the +room #conduit:fachschaften.org. From bd662b23e1e9244367a2cce1680f123c9146d0e6 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 10 Jul 2022 21:48:10 +0200 Subject: [PATCH 10/22] --typos --- .../contents.lr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr b/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr index 90cfe07..47b096e 100644 --- a/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr +++ b/content/u/blog/2022-07-10-launching-ipv6only-conduit-matrix-server/contents.lr @@ -41,7 +41,7 @@ How stable you ask? Well, ... ## Phasing in public beta server at ungleich -To give you an preview, we have setup an **IPv6 only** instance of +To give you a preview, we have setup an **IPv6 only** instance of conduit at [conduit.ungleich.ch](https://conduit.ungleich.ch). This server is open for public use, anyone (well, anyone with @@ -54,7 +54,7 @@ If you are looking for a production grade homeserver for yourself, have a look at our [hosted Matrix offerings](https://ungleich.ch/u/products/hosted-matrix-chat/). -That said - give it a shot, we think Conduit is a promising +That said - give it a shot. We think Conduit is a promising homeserver, especially given its focus on low resource usage. ## ... why IPv6 only? From 1899d5551c43078934ac545b91a3dbcd8ad12d3e Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 27 Aug 2022 18:37:42 +0200 Subject: [PATCH 11/22] ++blog for rook/ceph/nautilus --- .../contents.lr | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr new file mode 100644 index 0000000..d0ee04c --- /dev/null +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -0,0 +1,91 @@ +title: [WIP] Migrating Ceph Nautilus into Kubernetes + Rook +--- +pub_date: 2022-08-27 +--- +author: ungleich storage team +--- +twitter_handle: ungleich +--- +_hidden: no +--- +_discoverable: yes +--- +abstract: +How we move our Ceph clusters into kubernetes +--- +body: + +## Introduction + +At ungleich we are running multiple Ceph clusters. Some of them are +running Ceph Nautilus (14.x) based on +[Devuan](https://www.devuan.org/). Our newer Ceph Pacific (16.x) +clusters are running based on [Rook](https://rook.io/) on +[Kubernetes](https://kubernetes.io/) on top of +[Alpine Linux](https://alpinelinux.org/). + +In this blog article we will describe how to migrate +Ceph/Native/Devuan to Ceph/k8s+rook/Alpine Linux. + +## Work in Progress [WIP] + +This blog article is work in progress. The migration planning has +started, however the migration has not been finished yet. This article +will feature the different paths we take for the migration. + +## The Plan + +To continue operating the cluster during the migration, the following +steps are planned: + +* Setup a k8s cluster that can potentially communicate with the + existing ceph cluster +* Using the [disaster + recovery](https://rook.io/docs/rook/v1.9/Troubleshooting/disaster-recovery/) + guidelines from rook to modify the rook configuration to use the + previous fsid. +* Spin up ceph monitors and ceph managers in rook +* Retire existing monitors +* Shutdown a ceph OSD node, remove it's OS disk, boot it with Alpine + Linux +* Join the node into the k8s cluster +* Have rook pickup the existing disks and start the osds +* Repeat if successful +* Migrate to ceph pacific + +## Original cluster + +The target ceph cluster we want to migrate lives in the 2a0a:e5c0::/64 +network. Ceph is using: + +``` +public network = 2a0a:e5c0:0:0::/64 +cluster network = 2a0a:e5c0:0:0::/64 +``` + +## Kubernetes cluster networking inside the ceph network + +To be able to communicate with the existing OSDs, we will be using +sub networks of 2a0a:e5c0::/64 for kubernetes. As these networks +are part of the link assigned network 2a0a:e5c0::/64, we will use BGP +routing on the existing ceph nodes to create more specific routes into +the kubernetes cluster. + +As we plan to use either [cilium](https://cilium.io/) or +[calico](https://www.tigera.io/project-calico/) as the CNI, we can +configure kubernetes to directly BGP peer with the existing Ceph +nodes. + + +## Changelog + +### 2022-08-27 + +* The initial release of this blog article + + +## Follow up or questions + +You can join the discussion in the matrix room +#kubernetes:ungleich.ch about this migration. If don't have a matrix +account you can join using our chat on https://chat.with.ungleich.ch. From f5cf29913535542c755f3b643cbca8212fd2849d Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 27 Aug 2022 18:38:58 +0200 Subject: [PATCH 12/22] Fix k8s / matrix link --- .../contents.lr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index d0ee04c..0fb969d 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -86,6 +86,6 @@ nodes. ## Follow up or questions -You can join the discussion in the matrix room -#kubernetes:ungleich.ch about this migration. If don't have a matrix +You can join the discussion in the matrix room `#kubernetes:ungleich.ch` +about this migration. If don't have a matrix account you can join using our chat on https://chat.with.ungleich.ch. From e4c2bfd74c970ec777e80757ce55c24117c04701 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 27 Aug 2022 23:43:48 +0200 Subject: [PATCH 13/22] blog: +k8s migration / ceph --- .../contents.lr | 133 +++++++++++++++++- 1 file changed, 131 insertions(+), 2 deletions(-) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index 0fb969d..47e0fa6 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -53,7 +53,7 @@ steps are planned: * Repeat if successful * Migrate to ceph pacific -## Original cluster +### Original cluster The target ceph cluster we want to migrate lives in the 2a0a:e5c0::/64 network. Ceph is using: @@ -63,7 +63,7 @@ public network = 2a0a:e5c0:0:0::/64 cluster network = 2a0a:e5c0:0:0::/64 ``` -## Kubernetes cluster networking inside the ceph network +### Kubernetes cluster networking inside the ceph network To be able to communicate with the existing OSDs, we will be using sub networks of 2a0a:e5c0::/64 for kubernetes. As these networks @@ -76,6 +76,135 @@ As we plan to use either [cilium](https://cilium.io/) or configure kubernetes to directly BGP peer with the existing Ceph nodes. +## The setup + +### Kubernetes Bootstrap + +As usual we bootstrap 3 control plane nodes using kubeadm. The proxy +for the API resides in a different kuberentes cluster. + +We run + +``` +kubeadm init --config kubeadm.yaml +``` + +on the first node and join the other two control plane nodes. As +usual, joining the workers last. + +### k8s Networking / CNI + +For this setup we are using calico as described in the +[ungleich kubernetes +manual](https://redmine.ungleich.ch/projects/open-infrastructure/wiki/The_ungleich_kubernetes_infrastructure#section-23). + +``` +VERSION=v3.23.3 +helm repo add projectcalico https://docs.projectcalico.org/charts +helm upgrade --install --namespace tigera calico projectcalico/tigera-operator --version $VERSION --create-namespace +``` + +### BGP Networking on the old nodes + +To be able to import the BGP routes from Kubernetes, all old / native +hosts will run bird. The installation and configuration is as follows: + +``` +apt-get update +apt-get install -y bird2 + +router_id=$(hostname | sed 's/server//') + +cat > /etc/bird/bird.conf < 64 then accept; else reject; }; + export none; + }; +} +EOF +/etc/init.d/bird restart + +``` + +The router id must be adjusted for every host. As all hosts have a +unique number, we use that one as the router id. +The bird configuration allows to use dynamic peers so that any k8s +node in the network can peer with the old servers. + +We also use a filter to avoid receiving /64 routes, as they are +overlapping with the on link route. + +### BGP networking in Kubernetes + +Calico supports BGP peering and we use a rather standard calico +configuration: + +``` +--- +apiVersion: projectcalico.org/v3 +kind: BGPConfiguration +metadata: + name: default +spec: + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + asNumber: 65533 + serviceClusterIPs: + - cidr: 2a0a:e5c0:aaaa::/108 + serviceExternalIPs: + - cidr: 2a0a:e5c0:aaaa::/108 +``` + +Plus for each server and router we create a BGPPeer: + +``` +apiVersion: projectcalico.org/v3 +kind: BGPPeer +metadata: + name: serverXX +spec: + peerIP: 2a0a:e5c0::XX + asNumber: 65530 + keepOriginalNextHop: true +``` + +We apply the whole configuration using calicoctl: + +``` +./calicoctl create -f - < ~/vcs/k8s-config/bootstrap/p5-cow/calico-bgp.yaml +``` + +And a few seconds later we can observer the routes on the old / native +hosts: + +``` +bird> show protocols +Name Proto Table State Since Info +device1 Device --- up 23:09:01.393 +kernel1 Kernel master6 up 23:09:01.393 +k8s BGP --- start 23:09:01.393 Passive +k8s_1 BGP --- up 23:33:01.215 Established +k8s_2 BGP --- up 23:33:01.215 Established +k8s_3 BGP --- up 23:33:01.420 Established +k8s_4 BGP --- up 23:33:01.215 Established +k8s_5 BGP --- up 23:33:01.215 Established + +``` ## Changelog From 7c64ca00924cb32233dfdc9ce6e01815584c5cfc Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 27 Aug 2022 23:46:12 +0200 Subject: [PATCH 14/22] Minus yaml marker --- .../contents.lr | 1 - 1 file changed, 1 deletion(-) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index 47e0fa6..06bebf6 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -155,7 +155,6 @@ Calico supports BGP peering and we use a rather standard calico configuration: ``` ---- apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: From 393593c2c65490a04b3ec787e2846f2f10a0885d Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 28 Aug 2022 00:50:34 +0200 Subject: [PATCH 15/22] ++calico/net updates --- .../contents.lr | 95 ++++++++++++++++++- 1 file changed, 92 insertions(+), 3 deletions(-) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index 06bebf6..e42fc54 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -164,9 +164,9 @@ spec: nodeToNodeMeshEnabled: true asNumber: 65533 serviceClusterIPs: - - cidr: 2a0a:e5c0:aaaa::/108 + - cidr: 2a0a:e5c0:0:aaaa::/108 serviceExternalIPs: - - cidr: 2a0a:e5c0:aaaa::/108 + - cidr: 2a0a:e5c0:0:aaaa::/108 ``` Plus for each server and router we create a BGPPeer: @@ -205,12 +205,101 @@ k8s_5 BGP --- up 23:33:01.215 Established ``` +### Testing networking + +To verify that the new cluster is working properly, we can deploy a +tiny test deployment and see if it is globally reachable: + +``` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment +spec: + selector: + matchLabels: + app: nginx + replicas: 2 + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.20.0-alpine + ports: + - containerPort: 80 +--- +apiVersion: v1 +kind: Service +metadata: + name: nginx-service +spec: + selector: + app: nginx + ports: + - protocol: TCP + port: 80 +``` + +Using curl to access a sample service from the outside shows that +networking is working: + +``` +% curl -v http://[2a0a:e5c0:0:aaaa::e3c9] +* Trying 2a0a:e5c0:0:aaaa::e3c9:80... +* Connected to 2a0a:e5c0:0:aaaa::e3c9 (2a0a:e5c0:0:aaaa::e3c9) port 80 (#0) +> GET / HTTP/1.1 +> Host: [2a0a:e5c0:0:aaaa::e3c9] +> User-Agent: curl/7.84.0 +> Accept: */* +> +* Mark bundle as not supporting multiuse +< HTTP/1.1 200 OK +< Server: nginx/1.20.0 +< Date: Sat, 27 Aug 2022 22:35:49 GMT +< Content-Type: text/html +< Content-Length: 612 +< Last-Modified: Tue, 20 Apr 2021 16:11:05 GMT +< Connection: keep-alive +< ETag: "607efd19-264" +< Accept-Ranges: bytes +< + + + +Welcome to nginx! + + + +

Welcome to nginx!

+

If you see this page, the nginx web server is successfully installed and +working. Further configuration is required.

+ +

For online documentation and support please refer to +nginx.org.
+Commercial support is available at +nginx.com.

+ +

Thank you for using nginx.

+ + +* Connection #0 to host 2a0a:e5c0:0:aaaa::e3c9 left intact +``` + ## Changelog ### 2022-08-27 * The initial release of this blog article - +* Added k8s bootstrapping guide ## Follow up or questions From 827b24bfd80d6789790f01027f2d8c89ac488f46 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Mon, 29 Aug 2022 00:17:49 +0200 Subject: [PATCH 16/22] update for ceph cluster --- .../contents.lr | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index e42fc54..4e9e381 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -294,8 +294,31 @@ Commercial support is available at * Connection #0 to host 2a0a:e5c0:0:aaaa::e3c9 left intact ``` +So far we have found 1 issue: + +* Sometimes the old/native servers can reach the service, sometimes +they get a timeout + +In old calico notes on github it is referenced that overlapping +Pod/CIDR networks might be a problem. Additionally we cannot use +kubeadm to initialise the podsubnet to be a proper subnet of the node +subnet: + +``` +[00:15] server57.place5:~# kubeadm init --service-cidr 2a0a:e5c0:0:cccc::/108 --pod-network-cidr 2a0a:e5c0::/100 +I0829 00:16:38.659341 19400 version.go:255] remote version is much newer: v1.25.0; falling back to: stable-1.24 +podSubnet: Invalid value: "2a0a:e5c0::/100": the size of pod subnet with mask 100 is smaller than the size of node subnet with mask 64 +To see the stack trace of this error execute with --v=5 or higher +[00:16] server57.place5:~# +``` + + ## Changelog +### 2022-08-29 + +* Added kubernetes/kubeadm bootstrap issue + ### 2022-08-27 * The initial release of this blog article From c61af0220e1ef14724c79dd225ff5542fbbd0d0b Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 3 Sep 2022 15:33:15 +0200 Subject: [PATCH 17/22] ++blog --- .../contents.lr | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index 4e9e381..1de8aaf 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -312,9 +312,32 @@ To see the stack trace of this error execute with --v=5 or higher [00:16] server57.place5:~# ``` +### Networking 2022-09-03 + +* Instead of trying to merge the cluster networks, we will use + separate ranges +* According to the [ceph users mailing list + discussion](https://www.spinics.net/lists/ceph-users/msg73421.html) + it is actually not necessary for mons/osds to be in the same + network. In fact, we might be able to remove these settings + completely. + +So today we start with + +* podSubnet: 2a0a:e5c0:0:14::/64 +* serviceSubnet: 2a0a:e5c0:0:15::/108 + +Using BGP and calico, the kubernetes cluster is setup "as usual" (for +ungleich terms). + + ## Changelog +### 2022-09-03 + +* Next try starting for migration + ### 2022-08-29 * Added kubernetes/kubeadm bootstrap issue From 0c749903b06b83e0e4c88110e4e5f534662594f7 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 3 Sep 2022 15:34:14 +0200 Subject: [PATCH 18/22] --yaml separator --- .../contents.lr | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index 1de8aaf..fab492a 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -230,7 +230,11 @@ spec: image: nginx:1.20.0-alpine ports: - containerPort: 80 ---- +``` + +And the corresponding service: + +``` apiVersion: v1 kind: Service metadata: From bcc155c1fd9e81e6bb28aaa9d1587624454ccf1c Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 3 Sep 2022 16:39:55 +0200 Subject: [PATCH 19/22] Additional details for rook migration --- .../contents.lr | 125 ++++++++++++++++++ 1 file changed, 125 insertions(+) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index fab492a..dfda8ac 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -334,6 +334,131 @@ So today we start with Using BGP and calico, the kubernetes cluster is setup "as usual" (for ungleich terms). +### Ceph.conf change + +Originally our ceph.conf contained: + +``` +public network = 2a0a:e5c0:0:0::/64 +cluster network = 2a0a:e5c0:0:0::/64 +``` + +As of today they are removed and all daemons are restarted, allowing +the native cluster to speak with the kubernetes cluster. + +### Setting up rook + +Usually we deploy rook via argocd. However as we want to be easily +able to do manual intervention, we will first bootstrap rook via helm +directly and turn off various services + +``` +helm repo add rook https://charts.rook.io/release +helm repo update +``` + +We will use rook 1.8, as it is the last version to support Ceph +nautilus, which is our current ceph version. The latest 1.8 version is +1.8.10 at the moment. + +``` +helm upgrade --install --namespace rook-ceph --create-namespace --version v1.8.10 rook-ceph rook/rook-ceph +``` + +### Joining the 2 clusters, step 1: monitors and managers + +In the first step we want to add rook based monitors and managers +and replace the native ones. For rook to be able to talk to our +existing cluster, it needs to know + +* the current monitors/managers ("the monmap") +* the right keys to talk to the existing cluster +* the fsid + +As we are using v1.8, we will follow +[the guidelines for disaster recover of rook +1.8](https://www.rook.io/docs/rook/v1.8/ceph-disaster-recovery.html). + +Later we will need to create all the configurations so that rook knows +about the different pools. + +### Rook: CephCluster + +Rook has a configuration of type `CephCluster` that typically looks +something like this: + +``` +apiVersion: ceph.rook.io/v1 +kind: CephCluster +metadata: + name: rook-ceph + namespace: rook-ceph +spec: + cephVersion: + # see the "Cluster Settings" section below for more details on which image of ceph to run + image: quay.io/ceph/ceph:{{ .Chart.AppVersion }} + dataDirHostPath: /var/lib/rook + mon: + count: 5 + allowMultiplePerNode: false + storage: + useAllNodes: true + useAllDevices: true + onlyApplyOSDPlacement: false + mgr: + count: 1 + modules: + - name: pg_autoscaler + enabled: true + network: + ipFamily: "IPv6" + dualStack: false + crashCollector: + disable: false + # Uncomment daysToRetain to prune ceph crash entries older than the + # specified number of days. + daysToRetain: 30 +``` + +For migrating, we don't want rook in the first stage to create any +OSDs. So we will replace `useAllNodes: true` with `useAllNodes: false` +and `useAllDevices: true` also with `useAllDevices: false`. + +### Extracting a monmap + +To get access to the existing monmap, we can export it from the native +cluster using `ceph-mon -i {mon-id} --extract-monmap {map-path}`. +More details can be found on the [documentation for adding and +removing ceph +monitors](https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/). + + + +### Rook and Ceph pools + +Rook uses `CephBlockPool` to describe ceph pools as follows: + +``` +apiVersion: ceph.rook.io/v1 +kind: CephBlockPool +metadata: + name: hdd + namespace: rook-ceph +spec: + failureDomain: host + replicated: + size: 3 + deviceClass: hdd +``` + +In this particular cluster we have 2 pools: + +- one (ssd based, device class = ssd) +- hdd (hdd based, device class = hdd-big) + +The device class "hdd-big" is specific to this cluster as it used to +contain 2.5" and 3.5" HDDs in different pools. + ## Changelog From 6e69a121de3b002cf4d1160fd8853acd71fb8faa Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 3 Sep 2022 17:11:14 +0200 Subject: [PATCH 20/22] rook: defining the first step --- .../contents.lr | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index dfda8ac..1b30a47 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -424,6 +424,7 @@ For migrating, we don't want rook in the first stage to create any OSDs. So we will replace `useAllNodes: true` with `useAllNodes: false` and `useAllDevices: true` also with `useAllDevices: false`. + ### Extracting a monmap To get access to the existing monmap, we can export it from the native @@ -459,6 +460,71 @@ In this particular cluster we have 2 pools: The device class "hdd-big" is specific to this cluster as it used to contain 2.5" and 3.5" HDDs in different pools. +### [old] Analysing the ceph cluster configuration + +Taking the view from the old cluster, the following items are +important for adding new services/nodes: + +* We have a specific fsid that needs to be known + * The expectation would be to find that fsid in a configmap/secret in rook +* We have a list of running monitors + * This is part of the monmap and ceph.conf + * ceph.conf is used for finding the initial contact point + * Afterwards the information is provided by the monitors + * For rook it would be expected to have a configmap/secret listing + the current monitors +* The native clusters have a "ceph.client.admin.keyring" deployed which +allows adding and removing resources. + * Rook probably has a secret for keyrings + * Maybe multiple depending on how services are organised + +### Analysing the rook configurations + +Taking the opposite view, we can also checkout a running rook cluster + +### Configuring ceph after the operator deployment + +As soon as the operator and the crds have been deployed, we deploy the +following +[CephCluster](https://rook.io/docs/rook/v1.8/ceph-cluster-crd.html) +configuration: + +``` +apiVersion: ceph.rook.io/v1 +kind: CephCluster +metadata: + name: rook-ceph + namespace: rook-ceph +spec: + cephVersion: + image: quay.io/ceph/ceph:v14.2.21 + dataDirHostPath: /var/lib/rook + mon: + count: 5 + allowMultiplePerNode: false + storage: + useAllNodes: false + useAllDevices: false + onlyApplyOSDPlacement: false + mgr: + count: 1 + modules: + - name: pg_autoscaler + enabled: true + network: + ipFamily: "IPv6" + dualStack: false + crashCollector: + disable: false + # Uncomment daysToRetain to prune ceph crash entries older than the + # specified number of days. + daysToRetain: 30 +``` + +We wait for the cluster to initialise and stabilise before applying +changes. Important to note is that we use the ceph image version +v14.2.21, which is the same version as the native cluster. + ## Changelog @@ -466,6 +532,7 @@ contain 2.5" and 3.5" HDDs in different pools. ### 2022-09-03 * Next try starting for migration +* Looking deeper into configurations ### 2022-08-29 From 02608ea8fa517282ff76b1a6fd70cb6bb04a19f0 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sat, 3 Sep 2022 17:14:13 +0200 Subject: [PATCH 21/22] rook: add ceph-toolbox --- .../contents.lr | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index 1b30a47..fd39cfb 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -526,6 +526,73 @@ changes. Important to note is that we use the ceph image version v14.2.21, which is the same version as the native cluster. +### The ceph-toolbox + +To be able to view the current cluster status, we also deploy the +ceph-toolbox for interacting with rook: + +``` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: rook-ceph-tools + namespace: rook-ceph # namespace:cluster + labels: + app: rook-ceph-tools +spec: + replicas: 1 + selector: + matchLabels: + app: rook-ceph-tools + template: + metadata: + labels: + app: rook-ceph-tools + spec: + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: rook-ceph-tools + image: rook/ceph:v1.8.10 + command: ["/bin/bash"] + args: ["-m", "-c", "/usr/local/bin/toolbox.sh"] + imagePullPolicy: IfNotPresent + tty: true + securityContext: + runAsNonRoot: true + runAsUser: 2016 + runAsGroup: 2016 + env: + - name: ROOK_CEPH_USERNAME + valueFrom: + secretKeyRef: + name: rook-ceph-mon + key: ceph-username + - name: ROOK_CEPH_SECRET + valueFrom: + secretKeyRef: + name: rook-ceph-mon + key: ceph-secret + volumeMounts: + - mountPath: /etc/ceph + name: ceph-config + - name: mon-endpoint-volume + mountPath: /etc/rook + volumes: + - name: mon-endpoint-volume + configMap: + name: rook-ceph-mon-endpoints + items: + - key: data + path: mon-endpoints + - name: ceph-config + emptyDir: {} + tolerations: + - key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 5 +``` + ## Changelog From 3ce72e805fa22e8093ff55c33b657b8ce3b8b1a9 Mon Sep 17 00:00:00 2001 From: Nico Schottelius Date: Sun, 4 Sep 2022 00:55:02 +0200 Subject: [PATCH 22/22] ++rook notes --- .../contents.lr | 515 +++++++++++++++++- 1 file changed, 514 insertions(+), 1 deletion(-) diff --git a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr index fd39cfb..896c395 100644 --- a/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr +++ b/content/u/blog/2022-08-27-migrating-ceph-nautilus-into-kubernetes-with-rook/contents.lr @@ -460,6 +460,7 @@ In this particular cluster we have 2 pools: The device class "hdd-big" is specific to this cluster as it used to contain 2.5" and 3.5" HDDs in different pools. + ### [old] Analysing the ceph cluster configuration Taking the view from the old cluster, the following items are @@ -481,6 +482,77 @@ allows adding and removing resources. ### Analysing the rook configurations Taking the opposite view, we can also checkout a running rook cluster +and the rook disaster recovery documentation to identify what to +modify. + +Let's have a look at the secrets first: + +``` +cluster-peer-token-rook-ceph kubernetes.io/rook 2 320d +default-token-xm9xs kubernetes.io/service-account-token 3 320d +rook-ceph-admin-keyring kubernetes.io/rook 1 320d +rook-ceph-admission-controller kubernetes.io/tls 3 29d +rook-ceph-cmd-reporter-token-5mh88 kubernetes.io/service-account-token 3 320d +rook-ceph-config kubernetes.io/rook 2 320d +rook-ceph-crash-collector-keyring kubernetes.io/rook 1 320d +rook-ceph-mgr-a-keyring kubernetes.io/rook 1 320d +rook-ceph-mgr-b-keyring kubernetes.io/rook 1 320d +rook-ceph-mgr-token-ktt2m kubernetes.io/service-account-token 3 320d +rook-ceph-mon kubernetes.io/rook 4 320d +rook-ceph-mons-keyring kubernetes.io/rook 1 320d +rook-ceph-osd-token-8m6lb kubernetes.io/service-account-token 3 320d +rook-ceph-purge-osd-token-hznnk kubernetes.io/service-account-token 3 320d +rook-ceph-rgw-token-wlzbc kubernetes.io/service-account-token 3 134d +rook-ceph-system-token-lxclf kubernetes.io/service-account-token 3 320d +rook-csi-cephfs-node kubernetes.io/rook 2 320d +rook-csi-cephfs-plugin-sa-token-hkq2g kubernetes.io/service-account-token 3 320d +rook-csi-cephfs-provisioner kubernetes.io/rook 2 320d +rook-csi-cephfs-provisioner-sa-token-tb78d kubernetes.io/service-account-token 3 320d +rook-csi-rbd-node kubernetes.io/rook 2 320d +rook-csi-rbd-plugin-sa-token-dhhq6 kubernetes.io/service-account-token 3 320d +rook-csi-rbd-provisioner kubernetes.io/rook 2 320d +rook-csi-rbd-provisioner-sa-token-lhr4l kubernetes.io/service-account-token 3 320d +``` + +TBC + +### Creating additional resources after the cluster is bootstrapped + +To let rook know what should be there, we already create the two +`CephBlockPool` instances that match the existing pools: + +```apiVersion: ceph.rook.io/v1 +kind: CephBlockPool +metadata: + name: one + namespace: rook-ceph +spec: + failureDomain: host + replicated: + size: 3 + deviceClass: ssd +``` + +And for the hdd based pool: + +``` +apiVersion: ceph.rook.io/v1 +kind: CephBlockPool +metadata: + name: hdd + namespace: rook-ceph +spec: + failureDomain: host + replicated: + size: 3 + deviceClass: hdd-big +``` + +Saving both of these in ceph-blockpools.yaml and applying it: + +``` +kubectl -n rook-ceph apply -f ceph-blockpools.yaml +``` ### Configuring ceph after the operator deployment @@ -526,6 +598,29 @@ changes. Important to note is that we use the ceph image version v14.2.21, which is the same version as the native cluster. + +### rook v1.8 is incompatible with ceph nautilus + +After deploying the rook operator, the following error message is +printed in its logs: + +``` +2022-09-03 15:14:03.543925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed the ceph version check: the version does not meet the minimum version "15.2.0-0 octopus" +``` + +So we need to downgrade to rook v1.7. Using `helm search repo +rook/rook-ceph --versions` we identify the latest usable version +should be `v1.7.11`. + +We start the downgrade process using + +``` +helm upgrade --install --namespace rook-ceph --create-namespace --version v1.7.11 rook-ceph rook/rook-ceph +``` + +After downgrading the operator is starting the canary monitors and +continues to bootstrap the cluster. + ### The ceph-toolbox To be able to view the current cluster status, we also deploy the @@ -552,7 +647,7 @@ spec: dnsPolicy: ClusterFirstWithHostNet containers: - name: rook-ceph-tools - image: rook/ceph:v1.8.10 + image: rook/ceph:v1.7.11 command: ["/bin/bash"] args: ["-m", "-c", "/usr/local/bin/toolbox.sh"] imagePullPolicy: IfNotPresent @@ -593,6 +688,424 @@ spec: tolerationSeconds: 5 ``` +### Checking the deployments + +After the rook-operator finished deploying, the following deployments +are visible in kubernetes: + +``` +[17:25] blind:~% kubectl -n rook-ceph get deployment +NAME READY UP-TO-DATE AVAILABLE AGE +csi-cephfsplugin-provisioner 2/2 2 2 21m +csi-rbdplugin-provisioner 2/2 2 2 21m +rook-ceph-crashcollector-server48 1/1 1 1 2m3s +rook-ceph-crashcollector-server52 1/1 1 1 2m24s +rook-ceph-crashcollector-server53 1/1 1 1 2m2s +rook-ceph-crashcollector-server56 1/1 1 1 2m17s +rook-ceph-crashcollector-server57 1/1 1 1 2m1s +rook-ceph-mgr-a 1/1 1 1 2m3s +rook-ceph-mon-a 1/1 1 1 10m +rook-ceph-mon-b 1/1 1 1 8m3s +rook-ceph-mon-c 1/1 1 1 5m55s +rook-ceph-mon-d 1/1 1 1 5m33s +rook-ceph-mon-e 1/1 1 1 4m32s +rook-ceph-operator 1/1 1 1 102m +rook-ceph-tools 1/1 1 1 17m +``` + +Relevant for us are the mgr, mon and operator. To stop the cluster, we +will shutdown the deployments in the following order: + +* rook-ceph-operator: to prevent deployments to recover + +### Data / configuration comparison + +Logging into a host that is running mon-a, we find the following data +in it: + +``` +[17:36] server56.place5:/var/lib/rook# find +. +./mon-a +./mon-a/data +./mon-a/data/keyring +./mon-a/data/min_mon_release +./mon-a/data/store.db +./mon-a/data/store.db/LOCK +./mon-a/data/store.db/000006.log +./mon-a/data/store.db/000004.sst +./mon-a/data/store.db/CURRENT +./mon-a/data/store.db/MANIFEST-000005 +./mon-a/data/store.db/OPTIONS-000008 +./mon-a/data/store.db/OPTIONS-000005 +./mon-a/data/store.db/IDENTITY +./mon-a/data/kv_backend +./rook-ceph +./rook-ceph/crash +./rook-ceph/crash/posted +./rook-ceph/log +``` + +Which is pretty similar to the native nodes: + +``` +[17:37:50] red3.place5:/var/lib/ceph/mon/ceph-red3# find +. +./sysvinit +./keyring +./min_mon_release +./kv_backend +./store.db +./store.db/1959645.sst +./store.db/1959800.sst +./store.db/OPTIONS-3617174 +./store.db/2056973.sst +./store.db/3617348.sst +./store.db/OPTIONS-3599785 +./store.db/MANIFEST-3617171 +./store.db/1959695.sst +./store.db/CURRENT +./store.db/LOCK +./store.db/2524598.sst +./store.db/IDENTITY +./store.db/1959580.sst +./store.db/2514570.sst +./store.db/1959831.sst +./store.db/3617346.log +./store.db/2511347.sst +``` + +### Checking how monitors are created on native ceph + +To prepare for the migration we take 1 step back and verify how +monitors are created in the native cluster. The script used for +monitoring creation can be found on +[code.ungleich.ch](https://code.ungleich.ch/ungleich-public/ungleich-tools/src/branch/master/ceph/ceph-mon-create-start) +and contains the following logic: + +* get "mon." key +* get the monmap +* Run ceph-mon --mkfs using the monmap and keyring +* Start it + +In theory we could re-use these steps on a rook deployed monitor to +join our existing cluster. + +### Checking the toolbox and monitor pods for migration + +When the ceph-toolbox is deployed, we get a ceph.conf and a keyring in +/ect/ceph. The keyring is actually the admin keyring and allows us to +make modifications to the ceph cluster. The ceph.conf points to the +monitors and does not contain an fsid. + +The ceph-toolbox gets this informatoin via 1 configmap +("rook-ceph-mon-endpoints") and a secret ("rook-ceph-mon"). + +The monitor pods on the other hand have an empty ceph.conf and no +admin keyring deployed. + +### Try 1: recreating a monitor inside the existing cluster + +Let's try to reuse an existing monitor and join it into the existing +cluster. For this we will first shut down the rook-operator, to +prevent it to intefere with our migration. Then +modify the relevant configmaps and secrets and import the settings +from the native cluster. + +Lastly we will patch one of the monitor pods, inject the monmap from +the native cluster and then restart it. + +Let's give it a try. First we shutdown the rook-ceph-operator: + +``` +% kubectl -n rook-ceph scale --replicas=0 deploy/rook-ceph-operator +deployment.apps/rook-ceph-operator scaled +``` + +Then we patch the mon deployments to not run a monitor, but only +sleep: + +``` +for mon in a b c d e; do +kubectl -n rook-ceph patch deployment rook-ceph-mon-${mon} -p \ +'{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}'; + +kubectl -n rook-ceph patch deployment rook-ceph-mon-$mon --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' +done +``` + +No the pod is restarted and when we execute into it, we will see that +no monitor is running in it: + +``` +% kubectl -n rook-ceph exec -ti rook-ceph-mon-a-c9f8f554b-2fkhm -- sh +Defaulted container "mon" out of: mon, chown-container-data-dir (init), init-mon-fs (init) +sh-4.2# ps aux +USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND +root 1 0.0 0.0 4384 664 ? Ss 19:44 0:00 sleep infinity +root 7 0.0 0.0 11844 2844 pts/0 Ss 19:44 0:00 sh +root 13 0.0 0.0 51752 3384 pts/0 R+ 19:44 0:00 ps aux +sh-4.2# +``` + +Now for this pod to work with our existing cluster, we want to import +the monmap and join the monitor to the native cluster. As with any +mon, the data is stored below `/var/lib/ceph/mon/ceph-a/`. + +Before importing the monmap, let's have a look at the different rook +configurations that influence the ceph components + +### Looking at the ConfigMap in detail: rook-ceph-mon-endpoints + +As the name says, it contains the list of monitor endpoints: + +``` +kubectl -n rook-ceph edit configmap rook-ceph-mon-endpoints +... + + csi-cluster-config-json: '[{"clusterID":"rook-ceph","monitors":["[2a0a:e5c0:0:15::fc2]:6789"... + data: b=[2a0a:e5c0:0:15::9cd9]:6789,.... + mapping: '{"node":{"a":{"Name":"server56","Hostname":"server56","Address":"2a0a:e5c0::... +``` + +As eventually we want the cluster and csi to use the in-cluster +monitors, we don't need to modify it right away. + +### Looking at Secrets in detail: rook-ceph-admin-keyring + +The first interesting secret is **rook-ceph-admin-keyring**, which +contains the admin keyring. The old one of course, so we can edit this +secret and replace it with the client.admin secret from our native +cluster. + +We encode the original admin keyring using: + +``` +cat ceph.client.admin.keyring | base64 -w 0; echo "" +``` + +And then we update the secret it: + +``` +kubectl -n rook-ceph edit secret rook-ceph-admin-keyring +``` + +[done] + +### Looking at Secrets in detail: rook-ceph-config + +This secret contains two keys, **mon_host** and +**mon_initial_members**. The **mon_host** is a list of monitor +addresses. The **mon_host** only contains the monitor names, a, b, c, d and e. + +The environment variable **ROOK_CEPH_MON_HOST** in the monitor +deployment is set to to **mon_host** key of that secret, so monitors +will read from it. + + + +### Looking at Secrets in detail: rook-ceph-mon + +This secret contains the following interesting keys: + +* ceph-secret: the admin key (just the base64 key no section around + it) [done] +* ceph-username: "client.admin" +* fsid: the ceph cluster fsid +* mon-secret: The key of the [mon.] section + +It's important to mention to use `echo -n` when inserting +the keys or fsids. + +[done] + +### Looking at Secrets in detail: rook-ceph-mons-keyring + +Contains the key "keyring" containing the [mon.] and [client.admin] +sections: + + +``` +[mon.] + key = ... + +[client.admin] + key = ... + caps mds = "allow" + caps mgr = "allow *" + caps mon = "allow *" + caps osd = "allow *" +``` + +Using `base64 -w0 < ~/mon-and-client`. + +[done] + +### Importing the monmap + +Getting the current monmap from the native cluster: + +``` +ceph mon getmap -o monmap-20220903 + +scp root@old-monitor:monmap-20220903 +``` + +Adding it into the mon pod: + +``` +kubectl cp monmap-20220903 rook-ceph/rook-ceph-mon-a-6c46d4694-kxm5h:/tmp +``` + +Moving the old mon db away: + +``` +cd /var/lib/ceph/mon/ceph-a +mkdir _old +mv [a-z]* _old/ +``` + +Recreating the mon fails, as the volume is mounted directly onto it: + +``` +% ceph-mon -i a --mkfs --monmap /tmp/monmap-20220903 --keyring /tmp/mon-key +2022-09-03 21:44:48.268 7f1a738f51c0 -1 '/var/lib/ceph/mon/ceph-a' already exists and is not empty: monitor may already exist + +% mount | grep ceph-a +/dev/sda1 on /var/lib/ceph/mon/ceph-a type ext4 (rw,relatime) + +``` + +We can workaround this by creating all monitors on pods with other +names. So we can create mon b to e on the mon-a pod and mon-a on any +other pod. + +On rook-ceph-mon-a: + +``` +for mon in b c d e; +do ceph-mon -i $mon --mkfs --monmap /tmp/monmap-20220903 --keyring /tmp/mon-key; +done +``` + +On rook-ceph-mon-b: + +``` +mon=a +ceph-mon -i $mon --mkfs --monmap /tmp/monmap-20220903 --keyring /tmp/mon-key +``` + +Then we export the newly created mon dbs: + +``` +for mon in b c d e; +do kubectl cp rook-ceph/rook-ceph-mon-a-6c46d4694-kxm5h:/var/lib/ceph/mon/ceph-$mon ceph-$mon; +done +``` + +``` +for mon in a; +do kubectl cp rook-ceph/rook-ceph-mon-b-57d888dd9f-w8jkh:/var/lib/ceph/mon/ceph-$mon ceph-$mon; +done +``` + +And finally we test it by importing the mondb to mon-a: + +``` +kubectl cp ceph-a +rook-ceph/rook-ceph-mon-a-6c46d4694-kxm5h:/var/lib/ceph/mon/ +``` + +And the other mons: + +``` +kubectl cp ceph-b rook-ceph/rook-ceph-mon-b-57d888dd9f-w8jkh:/var/lib/ceph/mon/ + +``` + +### Re-enabling the rook-operator + +As the deployment + +``` +kubectl -n rook-ceph scale --replicas=1 deploy/rook-ceph-operator +``` + +Operator sees them running (with a shell) + +``` +2022-09-03 22:29:26.725915 I | op-mon: mons running: [d e a b c] + +``` + +Triggering recreation: + +``` +% kubectl -n rook-ceph delete deployment rook-ceph-mon-a +deployment.apps "rook-ceph-mon-a" deleted +``` + +Connected successfully to the cluster: + +``` + + services: + mon: 6 daemons, quorum red1,red2,red3,server4,server3,a (age 8s) + mgr: red3(active, since 8h), standbys: red2, red1, server4 + osd: 46 osds: 46 up, 46 in + +``` + +A bit later: + +``` + mon: 8 daemons, quorum (age 2w), out of quorum: red1, red2, red3, server4, server3, a, c, +d + mgr: red3(active, since 8h), standbys: red2, red1, server4 + osd: 46 osds: 46 up, 46 in + +``` + +And a little bit later also the mgr joined the cluster: + +``` + services: + mon: 8 daemons, quorum red2,red3,server4,server3,a,c,d,e (age 46s) + mgr: red3(active, since 9h), standbys: red1, server4, a, red2 + osd: 46 osds: 46 up, 46 in + +``` + +And a few minutes later all mons joined successfully: + +``` + mon: 8 daemons, quorum red3,server4,server3,a,c,d,e,b (age 31s) + mgr: red3(active, since 105s), standbys: red1, server4, a, red2 + osd: 46 osds: 46 up, 46 in + +``` + +We also need to ensure the toolbox is being updated/recreated: + +``` +kubectl -n rook-ceph delete pods rook-ceph-tools-5cf88dd58f-fwwlc +``` + +### Retiring the old monitors + + + +### The actual migration + +At this point we have 2 ceph clusters: + +* A new one in rook +* The old/native one + +The next steps are: + +Replace fsid in secrets/rook-ceph-mon with that of the old one. + ## Changelog