++blog: disk based booting
This commit is contained in:
Nico Schottelius
parent
54133b70ef
commit
9c45ac817d
1 changed files with 131 additions and 0 deletions
|
|
@ -0,0 +1,131 @@
|
||||||
|
title: Bye, bye netboot
|
||||||
|
---
|
||||||
|
pub_date: 2021-08-31
|
||||||
|
---
|
||||||
|
author: ungleich infrastructure team
|
||||||
|
---
|
||||||
|
twitter_handle: ungleich
|
||||||
|
---
|
||||||
|
_hidden: no
|
||||||
|
---
|
||||||
|
_discoverable: yes
|
||||||
|
---
|
||||||
|
abstract:
|
||||||
|
Data Center Light servers are switching to disk based boot
|
||||||
|
---
|
||||||
|
body:
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
|
||||||
|
Since the very beginning of the [Data Center Light
|
||||||
|
project](/u/projects/data-center-light) our servers have been
|
||||||
|
*somewhat stateless* and booted from their operating system from the
|
||||||
|
network.
|
||||||
|
|
||||||
|
From today on this changes and our servers are switched to boot from
|
||||||
|
an disk (SSD/NVMe/HDD). While this first seems counter intuitive with
|
||||||
|
growing a data center, let us explain why this makes sense for us.
|
||||||
|
|
||||||
|
## Netboot in a nutshell
|
||||||
|
|
||||||
|
There are different variants of how to netboot a server. In either
|
||||||
|
case, the server loads an executable from the network, typically via
|
||||||
|
TFTP or HTTP and then hands over execution to it.
|
||||||
|
|
||||||
|
The first option is to load the kernel and then later switch to an NFS
|
||||||
|
based filesystem. If the filesystem is read write, you usually need
|
||||||
|
one location per server or you mount it read only and possibly apply
|
||||||
|
an overlay for runtime configuration.
|
||||||
|
|
||||||
|
The second option is to load the kernel and an initramfs into memory
|
||||||
|
and stay inside the initramfs. The advantage of this approach is that
|
||||||
|
no NFS server is needed, but the whole operating system is inside the
|
||||||
|
memory.
|
||||||
|
|
||||||
|
The second option is what we used in Data Center Light for the last
|
||||||
|
couple of years.
|
||||||
|
|
||||||
|
## Netboot history at Data Center Light
|
||||||
|
|
||||||
|
Originally all our servers started with IPv4 PXE based
|
||||||
|
netboot. However as our data center is generally speaking IPv6 only,
|
||||||
|
the IPv4 DHCP+TFTP combination is an extra maintenance and also a
|
||||||
|
hindrance for network debugging: if you are in a single stack IPv6
|
||||||
|
only network, things are much easier to debug. No need to look for two
|
||||||
|
routing tables, no need to work around DHCP settings that might
|
||||||
|
interfere with what one wants to achieve via IPv6.
|
||||||
|
|
||||||
|
As the IPv4 addresses became more of a technical debt in our
|
||||||
|
infrastructure, we started flashing our network cards with
|
||||||
|
[ipxe](https://ipxe.org/), which allows even older network cards to
|
||||||
|
boot in IPv6 only networks.
|
||||||
|
|
||||||
|
Also in an IPv6 only netboot environment, it is easier to run
|
||||||
|
active-active routers, as hosts are not assigned DHCP leases. They
|
||||||
|
assign addresses themselves, which scales much nicer.
|
||||||
|
|
||||||
|
## Migrating away from netbooting
|
||||||
|
|
||||||
|
So why are we migrating away from netbooting, even after we migrated
|
||||||
|
to IPv6 only networking? There are multiple aspects:
|
||||||
|
|
||||||
|
On power failure, netbooted hosts lose their state. The operating
|
||||||
|
system that is loaded is the same for every server and needs some
|
||||||
|
configuration post-boot. We have solved this using
|
||||||
|
[cdist](https://www.cdi.st/), however the authentication-trigger
|
||||||
|
mechanism is non-trivial, if you want to keep your netboot images and
|
||||||
|
build steps public.
|
||||||
|
|
||||||
|
The second reason is state synchronisation: as we are having multiple
|
||||||
|
boot servers, we need to maintain the same state on multiple
|
||||||
|
machines. That is solvable via CI/CD pipelines, however the level of
|
||||||
|
automation on build servers is rather low, because the amount of OS
|
||||||
|
changes are low.
|
||||||
|
|
||||||
|
The third and main point is our ongoing migration towards
|
||||||
|
[kubernetes](https://kubernetes.io/). Originally our servers would
|
||||||
|
boot up, get configured for providing ceph storage or to be a
|
||||||
|
virtualisation host. The amount of binaries to keep in our in-memory
|
||||||
|
image was tiny, in the best case around 150MB. With the migration
|
||||||
|
towards kubernetes, every node is downloading the containers, which
|
||||||
|
can be comparable huge (gigabytes of data). The additional pivot_root
|
||||||
|
workarounds that are required for initramfs usage are just an
|
||||||
|
additional minor point that made us question our current setup.
|
||||||
|
|
||||||
|
## Automating disk based boot
|
||||||
|
|
||||||
|
We have servers from a variety of brands and each of them comes with a
|
||||||
|
variety of disk controllers: from simple pass-through SATA controllers
|
||||||
|
to full fledged hardware raid with onboard cache and battery for
|
||||||
|
protecting the cache - everything is in the mix.
|
||||||
|
|
||||||
|
So it is not easily possible to generate a stack of disks somewhere
|
||||||
|
and then add them, as the disk controller might add some (RAID0) meta
|
||||||
|
data to it.
|
||||||
|
|
||||||
|
To work around this problem, we insert the disk that is becoming the
|
||||||
|
boot disk in the future into the netbooted servers, install the
|
||||||
|
operating system from the running environment and at the next
|
||||||
|
maintenance window ensure that the server is actually booting from it.
|
||||||
|
|
||||||
|
If you are curious on how this works, you can checkout the script that
|
||||||
|
we use for
|
||||||
|
[Devuan/Debian](https://code.ungleich.ch/ungleich-public/ungleich-tools/-/blob/master/debian-devuan-install-on-disk.sh)
|
||||||
|
and
|
||||||
|
[Alpine Linux](https://code.ungleich.ch/ungleich-public/ungleich-tools/-/blob/master/alpine-install-on-disk.sh)
|
||||||
|
|
||||||
|
## The road continues
|
||||||
|
|
||||||
|
While a data center needs to be stable, it also needs to adapt to
|
||||||
|
newer technologies or different flows. The disk based boot is our
|
||||||
|
current solution for our path towards kubernetes migration, but who
|
||||||
|
knows - in the future things might look different again.
|
||||||
|
|
||||||
|
If you want to join the discussion, we have a
|
||||||
|
[Hacking and Learning
|
||||||
|
(#hacking-and-learning:ungleich.ch)](/u/projects/open-chat/) channel
|
||||||
|
on Matrix for an open exchange.
|
||||||
|
|
||||||
|
Oh and in case [you were wondering what we did
|
||||||
|
today](https://twitter.com/ungleich/status/1432627966316584968), we
|
||||||
|
switched to disk based booting ;-).
|
||||||
Loading…
Reference in a new issue