192 lines
6.1 KiB
Markdown
192 lines
6.1 KiB
Markdown
title: Building Ingress-less Kubernetes Clusters
|
|
---
|
|
pub_date: 2021-06-09
|
|
---
|
|
author: ungleich
|
|
---
|
|
twitter_handle: ungleich
|
|
---
|
|
_hidden: no
|
|
---
|
|
_discoverable: no
|
|
---
|
|
abstract:
|
|
|
|
---
|
|
body:
|
|
|
|
## Introduction
|
|
|
|
On [our journey to build and define IPv6 only kubernetes
|
|
clusters](https://www.nico.schottelius.org/blog/k8s-ipv6-only-cluster/)
|
|
we came accross some principles that seem awkward in the IPv6 only
|
|
world. Let us today have a look at the *LoadBalancer* and *Ingress*
|
|
concepts.
|
|
|
|
## Ingress
|
|
|
|
Let's have a look at the [Ingress
|
|
definition](https://kubernetes.io/docs/concepts/services-networking/ingress/)
|
|
definiton from the kubernetes website:
|
|
|
|
```
|
|
Ingress exposes HTTP and HTTPS routes from outside the cluster to
|
|
services within the cluster. Traffic routing is controlled by rules
|
|
defined on the Ingress resource.
|
|
```
|
|
|
|
So the ingress basically routes from outside to inside. But, in the
|
|
IPv6 world, services are already publicly reachable. It just
|
|
depends on your network policy.
|
|
|
|
## Services
|
|
|
|
Let's have a look at how services in IPv6 only clusters look like:
|
|
|
|
```
|
|
% kubectl get svc
|
|
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
etherpad ClusterIP 2a0a:e5c0:13:e2::a94b <none> 9001/TCP 19h
|
|
nginx-service ClusterIP 2a0a:e5c0:13:e2::3607 <none> 80/TCP 43h
|
|
postgres ClusterIP 2a0a:e5c0:13:e2::c9e0 <none> 5432/TCP 19h
|
|
...
|
|
```
|
|
All these services are world reachable, so a theorethical ingress does
|
|
actually not do any work. Services are routed to the pods as usual.
|
|
|
|
## ServiceTypes
|
|
|
|
While we are at looking at the k8s primitives, let's have a closer
|
|
look at the **Service**, specifically at 3 of the **ServiceTypes**
|
|
supported by k8s, including it's definition:
|
|
|
|
### ClusterIP
|
|
|
|
The k8s website says
|
|
|
|
```
|
|
Exposes the Service on a cluster-internal IP. Choosing this value
|
|
makes the Service only reachable from within the cluster. This is the
|
|
default ServiceType.
|
|
```
|
|
|
|
So in the context of IPv6, this sounds wrong. There is nothing that
|
|
makes an global IPv6 address be "internal", besides possible network
|
|
policies. The concept is probably coming from the strict difference of
|
|
RFC1918 space usually used in k8s clusters and not public IPv4.
|
|
|
|
This difference does not make a lot of sense in the IPv6 world though.
|
|
Seeing **services as public by default**, makes much more sense.
|
|
And simplifies your clusters a lot.
|
|
|
|
### NodePort
|
|
|
|
Let's first have a look at the definition again:
|
|
|
|
```
|
|
Exposes the Service on each Node's IP at a static port (the
|
|
NodePort). A ClusterIP Service, to which the NodePort Service routes,
|
|
is automatically created. You'll be able to contact the NodePort
|
|
Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.
|
|
```
|
|
|
|
Conceptually this can be similarily utilised in the IPv6 only world
|
|
like it does in the IPv4 world. However given that there are enough
|
|
addresses available with IPv6, this might not be such an interesting
|
|
ServiceType anymore.
|
|
|
|
|
|
### LoadBalancer
|
|
|
|
Before we have a look at this type, let's take some steps back
|
|
first to ...
|
|
|
|
|
|
## ... Load Balancing
|
|
|
|
There are a variety of possibilities to do load balancing. From simple
|
|
round robin, to ECMP based load balancing, to application aware,
|
|
potentially weighted load balancing.
|
|
|
|
So for load balancing, there is usually more than one solution and
|
|
there is likely not one size fits all.
|
|
|
|
So with this said, let.s have a look at the
|
|
**ServiceType LoadBalancer** definition:
|
|
|
|
```
|
|
Exposes the Service externally using a cloud provider's load
|
|
balancer. NodePort and ClusterIP Services, to which the external load
|
|
balancer routes, are automatically created.
|
|
```
|
|
|
|
So whatever the cloud provider offers, can be used, and that is a good
|
|
thing. However, let's have a look at how you get load balancing for
|
|
free in IPv6 only clusters:
|
|
|
|
## Load Balancing in IPv6 only clusters
|
|
|
|
So what is the most easy way of reliable load balancing in network?
|
|
[ECMP (equal cost multi path)](https://en.wikipedia.org/wiki/Equal-cost_multi-path_routing)
|
|
comes to the mind right away. Given that
|
|
kubernetes nodes can BGP peer with the network (upstream or the
|
|
switches), this basically gives load balancing to the world for free:
|
|
|
|
```
|
|
[ The Internet ]
|
|
|
|
|
[ k8s-node-1 ]-----------[ network ]-----------[ k8s-node-n]
|
|
[ ECMP ]
|
|
|
|
|
[ k8s-node-2]
|
|
|
|
```
|
|
|
|
In the real world on a bird based BGP upstream router
|
|
this looks as follows:
|
|
|
|
```
|
|
[18:13:02] red.place7:~# birdc show route
|
|
BIRD 2.0.7 ready.
|
|
Table master6:
|
|
...
|
|
2a0a:e5c0:13:e2::/108 unicast [place7-server1 2021-06-07] * (100) [AS65534i]
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 on eth0
|
|
unicast [place7-server4 2021-06-08] (100) [AS65534i]
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 on eth0
|
|
unicast [place7-server2 2021-06-07] (100) [AS65534i]
|
|
via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc on eth0
|
|
unicast [place7-server3 2021-06-07] (100) [AS65534i]
|
|
via 2a0a:e5c0:13:0:224:81ff:fee0:db7a on eth0
|
|
...
|
|
```
|
|
|
|
Which results into the following kernel route:
|
|
|
|
```
|
|
2a0a:e5c0:13:e2::/108 proto bird metric 32
|
|
nexthop via 2a0a:e5c0:13:0:224:81ff:fee0:db7a dev eth0 weight 1
|
|
nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 dev eth0 weight 1
|
|
nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 dev eth0 weight 1
|
|
nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc dev eth0 weight 1 pref medium
|
|
```
|
|
|
|
## TL;DR
|
|
|
|
We know, a TL;DR at the end is not the right thing to do, but hey, we
|
|
are at ungleich, aren't we?
|
|
|
|
In a nutshell, with IPv6 the concept of **Ingress**,
|
|
**Service** and the **LoadBalancer** ServiceType
|
|
types need to be revised, as IPv6 allows direct access without having
|
|
to jump through hoops.
|
|
|
|
If you are interesting in continuing the discussion,
|
|
we are there for you in
|
|
**the #hacking:ungleich.ch Matrix channel**
|
|
[you can signup here if you don't have an
|
|
account](https://chat.with.ungleich.ch).
|
|
|
|
Or if you are interested in an IPv6 only kubernetes cluster,
|
|
drop a mail to **support**-at-**ungleich.ch**.
|