title: Making kubernetes kube-dns publicly reachable
---
pub_date: 2021-06-13
---
author: ungleich
---
twitter_handle: ungleich
---
_hidden: no
---
_discoverable: yes
---
abstract:
Looking into IPv6 only DNS provided by kubernetes
---
body:

## Introduction

If you have seen our
[article about running kubernetes
Ingress-less](/u/blog/kubernetes-without-ingress/), you are aware that
we are pushing IPv6 only kubernetes clusters at ungleich.

Today, we are looking at making the "internal" kube-dns service world
reachable using IPv6 and global DNS servers.

## The kubernetes DNS service

If you have a look at your typical k8s cluster, you will notice that
you usually have two coredns pods running:

```
% kubectl -n kube-system get pods -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-558bd4d5db-gz5c7   1/1     Running   0          6d
coredns-558bd4d5db-hrzhz   1/1     Running   0          6d
```

These pods are usually served by the **kube-dns** service:

```
% kubectl -n kube-system get svc -l k8s-app=kube-dns
NAME       TYPE        CLUSTER-IP           EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   2a0a:e5c0:13:e2::a   <none>        53/UDP,53/TCP,9153/TCP   6d1h
```

As you can see, the kube-dns service is running on a publicly
reachable IPv6 address.

## IPv6 only DNS

IPv6 only DNS servers have one drawback: they cannot be reached via DNS
recursions, if the resolver is IPv4 only.

At [ungleich we run a variety of
services](https://redmine.ungleich.ch/projects/open-infrastructure/wiki)
to make IPv6 only services usable in the real world. In case of DNS,
we are using **DNS forwarders**. They are acting similar to HTTP
proxies, but for DNS.

So in our main DNS servers, dns1.ungleich.ch, dns2.ungleich.ch
and dns3.ungleich.ch we have added the following configuration:

```
zone "k8s.place7.ungleich.ch"  {
   type forward;
   forward only;
   forwarders { 2a0a:e5c0:13:e2::a; };
};
```

This tells the DNS servers to forward DNS queries that come in for
k8s.place7.ungleich.ch to **2a0a:e5c0:13:e2::a**.

Additionally we have added **DNS delegation** in the
place7.ungleich.ch zone:

```
k8s NS dns1.ungleich.ch.
k8s NS dns2.ungleich.ch.
k8s NS dns3.ungleich.ch.
```

## Using the kubernetes DNS service in the wild

With this configuration, we can now access IPv6 only
kubernetes services directly from the Internet. Let's first discover
the kube-dns service itself:

```
% dig kube-dns.kube-system.svc.k8s.place7.ungleich.ch. aaaa

; <<>> DiG 9.16.16 <<>> kube-dns.kube-system.svc.k8s.place7.ungleich.ch. aaaa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23274
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: f61925944f5218c9ac21e43960c64f254792e60f2b10f3f5 (good)
;; QUESTION SECTION:
;kube-dns.kube-system.svc.k8s.place7.ungleich.ch. IN AAAA

;; ANSWER SECTION:
kube-dns.kube-system.svc.k8s.place7.ungleich.ch. 27 IN AAAA 2a0a:e5c0:13:e2::a

;; AUTHORITY SECTION:
k8s.place7.ungleich.ch.	13	IN	NS	kube-dns.kube-system.svc.k8s.place7.ungleich.ch.
```

As you can see, the **kube-dns** service in the **kube-system**
namespace resolves to 2a0a:e5c0:13:e2::a, which is exactly what we
have configured.

At the moment, there is also an etherpad test service
named "ungleich-etherpad" running:

```
% kubectl get svc -l app=ungleichetherpad
NAME                TYPE        CLUSTER-IP              EXTERNAL-IP   PORT(S)    AGE
ungleich-etherpad   ClusterIP   2a0a:e5c0:13:e2::b7db   <none>        9001/TCP   3d19h
```

Let's first verify that it resolves:

```
% dig +short ungleich-etherpad.default.svc.k8s.place7.ungleich.ch aaaa
2a0a:e5c0:13:e2::b7db
```

And if that works, well, then we should also be able to access the
service itself!

```
% curl -I http://ungleich-etherpad.default.svc.k8s.place7.ungleich.ch:9001/
HTTP/1.1 200 OK
X-Powered-By: Express
X-UA-Compatible: IE=Edge,chrome=1
Referrer-Policy: same-origin
Content-Type: text/html; charset=utf-8
Content-Length: 6039
ETag: W/"1797-Dq3+mr7XP0PQshikMNRpm5RSkGA"
Set-Cookie: express_sid=s%3AZGKdDe3FN1v5UPcS-7rsZW7CeloPrQ7p.VaL1V0M4780TBm8bT9hPVQMWPX5Lcte%2BzotO9Lsejlk; Path=/; HttpOnly; SameSite=Lax
Date: Sun, 13 Jun 2021 18:36:23 GMT
Connection: keep-alive
Keep-Alive: timeout=5
```

(attention, this is a test service and might not be running when you
read this article at a later time)

## IPv6 vs. IPv4

Could we have achived the same with IPv4? The answere here is "maybe":
If the kubernetes service is reachable from globally reachable
nameservers via IPv4, then the answer is yes. This could be done via
public IPv4 addresses in the kubernetes cluster, via tunnels, VPNs,
etc.

However, generally speaking, the DNS service of a
kubernetes cluster running on RFC1918 IP addresses, is probably not
reachable from globally reachable DNS servers by default.

For IPv6 the case is a bit different: we are using globally reachable
IPv6 addresses in our k8s clusters, so they can potentially be
reachable without the need of any tunnel or whatsoever. Firewalling
and network policies can obviously prevent access, but if the IP
addresses are properly routed, they will be accessible from the public
Internet.

And this makes things much easier for DNS servers, which are also
having IPv6 connectivity.

The following pictures shows the practical difference between the two
approaches:

![](/u/image/k8s-v6-v4-dns.png)

## Does this make sense?

That clearly depends on your use-case. If you want your service DNS
records to be publicly accessible, then the clear answer is yes.

If your cluster services are intended to be internal only
(see [previous blog post](/u/blog/kubernetes-without-ingress/), then
exposing the DNS service to the world might not be the best option.

## Note on security

CoreDNS inside kubernetes is by default configured to allow resolving
for *any* client that can reach it. Thus if you make your kube-dns
service world reachable, you also turn it into an open resolver.

The following coredns configuration **does** correctly block
requests, **IF your coredns version is new enough**:


```
  Corefile: |
    .:53 {
        acl k8s.place7.ungleich.ch {
             allow net ::/0
        }
        acl . {
              allow net 2a0a:e5c0:13::/48
              block
        }
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
...
```

We tested this with
[coredns-1.8.4](https://github.com/coredns/coredns/issues/4697) in
which the ACL behaviour is fixed.

## More of this

We are discussing
kubernetes and IPv6 related topics in
**the #hacking:ungleich.ch Matrix channel**
([you can signup here if you don't have an
account](https://chat.with.ungleich.ch)) and will post more about our
k8s journey in this blog. Stay tuned!