ungleich-staticcms/content/u/blog/how-to-build-an-openstack-alternative-step-1/contents.lr

title: How to build an OpenStack alternative: Step 1, the prototype
---
pub_date: 2020-01-11
---
author: ungleich virtualisation team
---
twitter_handle: ungleich
---
_hidden: no
---
_discoverable: yes
---
abstract:
The step by step guide for doing it yourself
---
body:

In this article we describe a first

## Find out what you need

When we say building an alternative to OpenStack, we have something
specific in our mind. This might be different from what you think
OpenStack is for. For us it is running a lot of virtual machines for
customers with a lot of storage attached. With self service and
automated payments.

All code I refer to in this article can be found on
[code.ungleich.ch](https://code.ungleich.ch/uncloud/uncloud/tree/master/uncloud/hack/hackcloud).

## Creating a network

The current setup at [Data Center
Light](/u/projects/data-center-light) relies heavily on VLANs. VLANs
however have a similar problem as IPv4 addresses: there are not that
many of them. So for our Openstack replacement we decided to go with
[VXLANs](https://en.wikipedia.org/wiki/Virtual_Extensible_LAN)
instead. We also considered
[SRV6](https://www.segment-routing.net/tutorials/2017-12-05-srv6-introduction/),
however we did not see a advantage for our use case. In fact, VXLANs
seems to be much simpler.

So before running a VM, we create a new VXLAN device and add it to a
bridge. This roughly looks as follows:

```
netid=100
dev=eth0

vxlandev=vxlan${netid}
bridgedev=br${netid}

# Create the vxlan device
ip -6 link add ${vxlandev} type vxlan \
    id ${netid} \
    dstport 4789 \
    group ff05::${netid} \
    dev ${dev} \
    ttl 5

ip link set ${vxlandev} up

# Create the bridge
ip link add ${bridgedev} type bridge
ip link set ${bridgedev} up

# Add the vxlan device into the bridge
ip link set ${vxlandev} master ${bridgedev} up
```

As you can see, we are using IPv6 multicast underlying the VXLAN,
which is very practical in an IPv6 first data center.

## IP address management (IPAM)

Speaking of IPv6 first, all VMs in our new setup will again be IPv6
only and IPv4 addresses will be mapped to it via NAT64. This is very
similar to what you see at AWS, just that AWS uses
[RFC1918](https://tools.ietf.org/html/rfc1918) private IPv4 space
instead of [global unique IPv6
addresses](https://tools.ietf.org/html/rfc3587), which we do.

The advantage of using IPv6 here is that you will never ever have a
collision and that your VM is very clean: no need to think about IPv4
firewall rules, you only need to configure IPv6 settings.

In the IPv6 world, we use router advertisements as an alternative to
DHCP in the IPv4 world. This has the advantage that no state is kept
on the server.

To enable our IPAM, we add an IPv6 address to our bridge and enable
the radvd daemon:

```
ip addr add ${ip} dev ${bridgedev}
radvd -C ./radvd.conf  -n -p ./radvdpid
```

A sample radvd configuration we used for testing looks like this:

```
interface br100
{
  AdvSendAdvert on;
  MinRtrAdvInterval 3;
  MaxRtrAdvInterval 5;
  AdvDefaultLifetime 3600;

  prefix 2a0a:e5c1:111:888::/64 {
  };

  RDNSS 2a0a:e5c0::3 2a0a:e5c0::4 { AdvRDNSSLifetime 6000; };
  DNSSL place7.ungleich.ch {  AdvDNSSLLifetime 6000; } ;
};
```

With this, we are ready to spawn a VM!

## Create a VM

The current setup at Data Center Light uses libvirtd for creating
VMs. This is problematic, because libvirtd is not very reliabe:
sometimes it stops to answer `virsh` commands or begins to use 100%
CPU and needs to be killed and restarted regularly. We have seen this
behaviour on CentOS 5, CentOS 6, Debian 8 and Devuan 9.

So in our version, we skip libvirt and run qemu directly. It turns out
that this is actually not that hard and can be done using the
following script:


```
vmid=$1; shift

qemu=/usr/bin/qemu-system-x86_64

accel=kvm
#accel=tcg

memory=1024
cores=2
uuid=732e08c7-84f8-4d43-9571-263db4f80080

export bridge=br100

$qemu -name uc${vmid} \
      -machine pc,accel=${accel} \
      -m ${memory} \
      -smp ${cores} \
      -uuid ${uuid} \
      -drive file=alpine-virt-3.11.2-x86_64.iso,media=cdrom \
      -netdev tap,id=netmain,script=./ifup.sh \
      -device virtio-net-pci,netdev=netmain,id=net0,mac=02:00:f0:a9:c4:4e
```

This starts a VM with a hard coded mac address using KVM
acceleration. We give the VM 2 cores and assign it an UUID so that we
can easily find it again later. For testing, we have attached an
[Alpine Linux ISO](https://alpinelinux.org/).

The interesting part is however the network part. We create a virtio
based network card and execute `ifup.sh` after qemu has been started.

The ifup.sh script looks as follows:

```
dev=$1; shift

# bridge is setup from outside
ip link set dev "$dev" master ${bridge}
ip link set dev "$dev" up
```

It basically adds the tap device to the previously created bridge.

## That's all there is

Only using above steps we spawned a test VM on a test machine that is
reachable at `2a0a:e5c1:111:888:0:f0ff:fea9:c44e`, world wide. If our
test machine is on, you should be able to reach it from anywhere in
the world.

Obviously this is not a full OpenStack replacement. However we wanted
to share the small steps that we take for creating it. And we really
like running a virtual machine hosting and wanted to show you how
much fun it can be.

## Next step

A lot of things in the above example are hard code and aren't usable
for customers directly. In the next step we will generalise some of
the above functions to get more and more nearby to provide a fully
usable OpenStack alternative.

If you are interested in this topic, you can join us on the [ungleich
chat](https://chat.with.ungleich.ch), the full development of our
alternative is open source.