ungleich-staticcms/content/u/blog/kubernetes-without-ingress/contents.lr

title: Building Ingress-less Kubernetes Clusters
---
pub_date: 2021-06-09
---
author: ungleich
---
twitter_handle: ungleich
---
_hidden: no
---
_discoverable: no
---
abstract:

---
body:

## Introduction

On [our journey to build and define IPv6 only kubernetes
clusters](https://www.nico.schottelius.org/blog/k8s-ipv6-only-cluster/)
we came accross some principles that seem awkward in the IPv6 only
world. Let us today have a look at the *LoadBalancer* and *Ingress*
concepts.

## Ingress

Let's have a look at the [Ingress
definition](https://kubernetes.io/docs/concepts/services-networking/ingress/)
definiton from the kubernetes website:

```
Ingress exposes HTTP and HTTPS routes from outside the cluster to
services within the cluster. Traffic routing is controlled by rules
defined on the Ingress resource.
```

So the ingress basically routes from outside to inside. But, in the
IPv6 world, services are already publicly reachable. It just
depends on your network policy.

## Services

Let's have a look at how services in IPv6 only clusters look like:

```
% kubectl get svc
NAME                                      TYPE        CLUSTER-IP              EXTERNAL-IP   PORT(S)                      AGE
etherpad                                  ClusterIP   2a0a:e5c0:13:e2::a94b   <none>        9001/TCP                     19h
nginx-service                             ClusterIP   2a0a:e5c0:13:e2::3607   <none>        80/TCP                       43h
postgres                                  ClusterIP   2a0a:e5c0:13:e2::c9e0   <none>        5432/TCP                     19h
...
```
All these services are world reachable, so a theorethical ingress does
actually not do any work. Services are routed to the pods as usual.

## ServiceTypes

While we are at looking at the k8s primitives, let's have a closer
look at the **Service**, specifically at 3 of the **ServiceTypes**
supported by k8s, including it's definition:

### ClusterIP

The k8s website says

```
Exposes the Service on a cluster-internal IP. Choosing this value
makes the Service only reachable from within the cluster. This is the
default ServiceType.
```

So in the context of IPv6, this sounds wrong. There is nothing that
makes an global IPv6 address be "internal", besides possible network
policies. The concept is probably coming from the strict difference of
RFC1918 space usually used in k8s clusters and not public IPv4.

This difference does not make a lot of sense in the IPv6 world though.
Seeing **services as public by default**, makes much more sense.
And simplifies your clusters a lot.

### NodePort

Let's first have a look at the definition again:

```
Exposes the Service on each Node's IP at a static port (the
NodePort). A ClusterIP Service, to which the NodePort Service routes,
is automatically created. You'll be able to contact the NodePort
Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.
```

Conceptually this can be similarily utilised in the IPv6 only world
like it does in the IPv4 world. However given that there are enough
addresses available with IPv6, this might not be such an interesting
ServiceType anymore.


### LoadBalancer

Before we have a look at this type, let's take some steps back
first to ...


## ... Load Balancing

There are a variety of possibilities to do load balancing. From simple
round robin, to ECMP based load balancing, to application aware,
potentially weighted load balancing.

So for load balancing, there is usually more than one solution and
there is likely not one size fits all.

So with this said, let.s have a look at the
**ServiceType LoadBalancer** definition:

```
Exposes the Service externally using a cloud provider's load
balancer. NodePort and ClusterIP Services, to which the external load
balancer routes, are automatically created.
```

So whatever the cloud provider offers, can be used, and that is a good
thing. However, let's have a look at how you get load balancing for
free in IPv6 only clusters:

## Load Balancing in IPv6 only clusters

So what is the most easy way of reliable load balancing in network?
[ECMP (equal cost multi path)](https://en.wikipedia.org/wiki/Equal-cost_multi-path_routing)
comes to the mind right away. Given that
kubernetes nodes can BGP peer with the network (upstream or the
switches), this basically gives load balancing to the world for free:

```
                            [ The Internet ]
                                   |
     [ k8s-node-1 ]-----------[ network ]-----------[ k8s-node-n]
                              [  ECMP   ]
                                   |
                             [ k8s-node-2]

```

In the real world on a bird based BGP upstream router
this looks as follows:

```
[18:13:02] red.place7:~# birdc show route
BIRD 2.0.7 ready.
Table master6:
...
2a0a:e5c0:13:e2::/108 unicast [place7-server1 2021-06-07] * (100) [AS65534i]
	via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 on eth0
                     unicast [place7-server4 2021-06-08] (100) [AS65534i]
	via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 on eth0
                     unicast [place7-server2 2021-06-07] (100) [AS65534i]
	via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc on eth0
                     unicast [place7-server3 2021-06-07] (100) [AS65534i]
	via 2a0a:e5c0:13:0:224:81ff:fee0:db7a on eth0
...
```

Which results into the following kernel route:

```
2a0a:e5c0:13:e2::/108 proto bird metric 32
	nexthop via 2a0a:e5c0:13:0:224:81ff:fee0:db7a dev eth0 weight 1
	nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:3554 dev eth0 weight 1
	nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:3564 dev eth0 weight 1
	nexthop via 2a0a:e5c0:13:0:225:b3ff:fe20:38cc dev eth0 weight 1 pref medium
```

## TL;DR

We know, a TL;DR at the end is not the right thing to do, but hey, we
are at ungleich, aren't we?

In a nutshell, with IPv6 the concept of **Ingress**,
**Service** and the **LoadBalancer** ServiceType
types need to be revised, as IPv6 allows direct access without having
to jump through hoops.

If you are interesting in continuing the discussion,
we are there for you in
**the #hacking:ungleich.ch Matrix channel**
[you can signup here if you don't have an
account](https://chat.with.ungleich.ch).

Or if you are interested in an IPv6 only kubernetes cluster,
drop a mail to **support**-at-**ungleich.ch**.