++ three new blog articles

content/u/blog/datacenterlight-active-active-routing/contents.lr

@@ -0,0 +1,161 @@
+title: Active-Active Routing Paths in Data Center Light
+---
+pub_date: 2019-11-08
+---
+author: Nico Schottelius
+---
+twitter_handle: NicoSchottelius
+---
+_hidden: no
+---
+_discoverable: no
+---
+abstract:
+
+---
+body:
+
+From our last two blog articles (a, b) you probably already know that
+it is spring network cleanup in [Data Center Light](https://datacenterlight.ch).
+
+In [first blog article]() we described where we started and in
+the [second blog article]() you could see how we switched our
+infrastructure to IPv6 only netboot.
+
+In this article we will dive a bit more into the details of our
+network architecture and which problems we face with active-active
+routers.
+
+## Network architecture
+
+Let's have a look at a simplified (!) diagram of the network:
+
+... IMAGE
+
+Doesn't look that simple, does it? Let's break it down into small
+pieces.
+
+## Upstream routers
+
+We have a set of **upstream routers** which work stateless. They don't
+have any stateful firewall rules, so both of them can work actively
+without state synchronisation. Moreover, both of them peer with the
+data center upstreams. These are fast routers and besides forwarding,
+they also do **BGP peering** with our upstreams.
+
+Over all the upstream routers are very simple machines, mostly running
+bird and forwarding packets all day. They also provide a DNS service
+(resolving and authoritative), because they are always up and can
+announce service IPs via BGP or via OSPF to our network.
+
+## Internal routers
+
+The internal routers on the other hand provide **stateful routing**,
+**IP address assignments** and **netboot services**. They are a bit
+more complicated compared to the upstream routers, but they care only
+a small routing table.
+
+## Communication between the routers
+
+All routers employ OSPF and BGP for route exchange. Thus the two
+upstream routers learn about the internal networks (IPv6 only, as
+usual) from the internal routers.
+
+## Sessions
+
+Sessions in networking are almost always an evil. You need to store
+them (at high speed), you need to maintain them (updating, deleting)
+and if you run multiple routers, you even need to sychronise them.
+
+In our case the internal routers do have session handling, as they are
+providing a stateful firewall. As we are using a multi router setup,
+things can go really wrong if the wrong routes are being used.
+
+Let's have a look at this a bit more in detail.
+
+## The good path
+
+IMAGE2: good
+
+If a server sends out a packet via router1 and router1 eventually
+receives the answer, everything is fine. The returning packet matches
+the state entry that was created by the outgoing packet and the
+internal router forwards the packet.
+
+## The bad path
+
+IMAGE3: bad
+
+However if the
+
+## Routing paths
+
+If we want to go active-active routing, the server can choose between
+either internal router for sending out the packet. The internal
+routers again have two upstream routers. So with the return path
+included, the following paths exist for a packet:
+
+Outgoing paths:
+
+* servers->router1->upstream router1->internet
+* servers->router1->upstream router2->internet
+* servers->router2->upstream router1->internet
+* servers->router2->upstream router2->internet
+
+And the returning paths are:
+
+* internet->upstream router1->router 1->servers
+* internet->upstream router1->router 2->servers
+* internet->upstream router2->router 1->servers
+* internet->upstream router2->router 2->servers
+
+So on average, 50% of the routes will hit the right router on
+return. However servers as well as upstream routers are not using load
+balancing like ECMP, so once an incorrect path has been chosen, the
+packet loss is 100%.
+
+## Session synchronisation
+
+In the first article we talked a bit about keepalived and that
+it helps to operate routers in an active-passive mode. This did not
+turn out to be the most reliable method. Can we do better with
+active-active routers and session synchronisation?
+
+Linux supports this using
+[conntrackd](http://conntrack-tools.netfilter.org/). However,
+conntrackd supports active-active routers on a **flow based** level,
+but not on a **packet** based level. The difference is that the
+following will not work in active-active routers with conntrackd:
+
+```
+#1 Packet (in the original direction) updates state in Router R1 ->
+   submit state to R2
+#2 Packet (in the reply direction) arrive to Router R2 before state
+   coming from R1 has been digested.
+
+With strict stateful filtering, Packet #2 will be dropped and it will
+trigger a retransmission.
+```
+(quote from Pablo Neira Ayuso, see below for more details)
+
+Some of you will mumble something like **latency** in their head right
+now. If the return packet is guaranteed to arrive after state
+synchronisation, then everything is fine, However, if the reply is
+faster than the state synchronisation, packets will get dropped.
+
+In reality, this will work for packets coming and going to the
+Internet. However, in our setup the upstream routers are route between
+different data center locations, which are in the sub micro second
+latency area - i.e. lan speed, because they are interconnected with
+dark fiber links.
+
+
+## Take away
+
+Before moving on to the next blog article, we would like to express
+our thanks to Pablo Neira Ayuso, who gave very important input for
+session based firewalls and session synchronisation.
+
+So active-active routing seems not to have a straight forward
+solution. Read in the [next blog article](/) on how we solved the
+challenge in the end.