title: Building Ingress-less Kubernetes Clusters --- pub_date: 2021-06-09 --- author: ungleich --- twitter_handle: ungleich --- _hidden: no --- _discoverable: no --- abstract: --- body: ## Introduction On [our journey to build and define IPv6 only kubernetes clusters](https://www.nico.schottelius.org/blog/k8s-ipv6-only-cluster/) we came accross some principles that seem awkward in the IPv6 only world. Let us today have a look at the *LoadBalancer* and *Ingress* concepts. ## Ingress Let's have a look at the [Ingress definition](https://kubernetes.io/docs/concepts/services-networking/ingress/) definiton from the kubernetes website: ``` Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. ``` So the ingress basically routes from outside to inside. But, in the IPv6 world, services are already publicly reachable. It just depends on your network policy. ## Services Let's have a look at how services in IPv6 only clusters look like: ``` % kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE etherpad ClusterIP 2a0a:e5c0:13:e2::a94b 9001/TCP 19h nginx-service ClusterIP 2a0a:e5c0:13:e2::3607 80/TCP 43h postgres ClusterIP 2a0a:e5c0:13:e2::c9e0 5432/TCP 19h ... ``` All these services are world reachable, so a theorethical ingress does actually not do any work. Services are routed to the pods as usual. ## ServiceTypes While we are at looking at the k8s primitives, let's have a closer look at the **Service**, specifically at 3 of the **ServiceTypes** supported by k8s, including it's definition: ### ClusterIP The k8s website says ``` Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType. ``` So in the context of IPv6, this sounds wrong. There is nothing that makes an global IPv6 address be "internal", besides possible network policies. The concept is probably coming from the strict difference of RFC1918 space usually used in k8s clusters and not public IPv4. This difference does not make a lot of sense in the IPv6 world though. Seeing **services as public by default**, makes much more sense. And simplifies your clusters a lot. ### NodePort Let's first have a look at the definition again: ``` Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting :. ``` Conceptually this can be similarily utilised in the IPv6 only world like it does in the IPv4 world. However given that there are enough addresses available with IPv6, this might not be such an interesting ServiceType anymore. ### LoadBalancer Before we have a look at this type, let's take some steps back first to ... ## ... Load Balancing There are a variety of possibilities to do load balancing. From simple round robin, to ECMP based load balancing, to application aware, potentially weighted load balancing. So for load balancing, there is usually more than one solution and there is likely not one size fits all. So with this said, let.s have a look at the **ServiceType LoadBalancer** definition: ``` Exposes the Service externally using a cloud provider's load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created. ``` So whatever the cloud provider offers, can be used, and that is a good thing. However, let's have a look at how you get load balancing for free in IPv6 only clusters: ## Load Balancing in IPv6 only clusters So what is the most easy way of reliable load balancing in network? [ECMP (equal cost multi path)](https://en.wikipedia.org/wiki/Equal-cost_multi-path_routing) comes to the mind right away. Given that kubernetes nodes can BGP peer with the network (upstream or the switches), this basically gives load balancing to the world for free: ``` [ The Internet ] | [ k8s-node-1 ]-----------[ network ]-----------[ k8s-node-n] [ ECMP ] | [ k8s-node-2] ``` In the real world on a bird based BGP upstream router this looks as follows: ``` [18:13:02] red.place7:~# birdc show route BIRD 2.0.7 ready. Table master6: ... 2a0a:e5c0:13:e2::/108 unicast [place7-server1 2021-06-07] * (100) [AS65534i] via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 on eth0 unicast [place7-server4 2021-06-08] (100) [AS65534i] via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 on eth0 unicast [place7-server2 2021-06-07] (100) [AS65534i] via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc on eth0 unicast [place7-server3 2021-06-07] (100) [AS65534i] via 2a0a:e5c0:13:0:224:81ff:fee0:db7a on eth0 ... ``` Which results into the following kernel route: ``` 2a0a:e5c0:13:e2::/108 proto bird metric 32 nexthop via 2a0a:e5c0:13:0:224:81ff:fee0:db7a dev eth0 weight 1 nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 dev eth0 weight 1 nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 dev eth0 weight 1 nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc dev eth0 weight 1 pref medium ``` ## TL;DR We know, a TL;DR at the end is not the right thing to do, but hey, we are at ungleich, aren't we? In a nutshell, with IPv6 the concept of **Ingress**, **Service** and the **LoadBalancer** ServiceType types need to be revised, as IPv6 allows direct access without having to jump through hoops. If you are interesting in continuing the discussion, we are there for you in **the #hacking:ungleich.ch Matrix channel** [you can signup here if you don't have an account](https://chat.with.ungleich.ch). Or if you are interested in an IPv6 only kubernetes cluster, drop a mail to **support**-at-**ungleich.ch**.