k8s_gateway: k8s-gateway breaks ability of internal systems to resolve external domain names

when deploying the following, all future attempts to resolve any other dns queries fails.

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease

metadata:
  name: excoredns
  namespace: kube-system

spec:
    interval: 5m
    install:
        createNamespace: true

    chart:
        spec:
            chart: k8s-gateway
            version: 1.1.9
            interval: 5m
            sourceRef:
                kind: HelmRepository
                name: k8s-gateway-helm-repo
                namespace: flux-system

    values:
      domain: kube.lan
      watchedResources: [ 'Ingress' ]
      fallthrough:
        enabled: true

ip address of my one and only k3s node is 192.168.8.222
in my openwrt router, i’ve added to the dnsmasq settings a dns-fowarding of /kube.lan/192.168.8.222/

if i add the following :

      extraZonePlugins: |
        forward . 1.1.1.1 1.0.0.1 {
          tls_servername cloudflare-dns.com
        }

Then dns queries for external hostnames are able to leave my local network and get resolved, but then so does every other query for the cluster.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 18 (1 by maintainers)

Most upvoted comments

👍🏻 If i disable systemd-resolved and wipe k3s then restart, it all works.

I imagine this isn’t usually a problem for people running docker via DockerForWin, or DockerForMac or some cloud-init based linux os that doesn’t setup resolveconf.

#mysterysolved

airtonix on Sep 23, 2022

as it applies, something interfers with the ability of the cluster to pull docker images. the error looks like it can’t resolve dns queries any more.

This means your k8s/k3s node DNS resolver is misconfigured.

dns queries to the outside world fail to resolve for anything. so now the rest of the fluxcd process throws errors everywhere because it can’t pull other docker images.

Seems like a symptom of the same problem.

Just so that we’re clear, our plugin, k8s_gateway, is designed to resolve external k8s resources (ingress, services etc.). You can combine it with other plugins, like forward if you also want it to act as a DNS forwarder, but this is entirely optional. For all intra-cluster DNS needs, you have to use standard kubernetes DNS add-on (coredns with kubernetes plugin).

Here are some of my thoughts based on what I understood from your description:

i only want dns queries for *.kube.lan to be handled by 192.168.8.222 i do not want any non *.kube.lan dns queries from the rest of my lan to be sent to 192.168.8.222

This should be the responsibility of dnsmasq/openwrt resolver.

i do not want dns queries for any internal cluster domain names to be leaving the cluster. i want queries for *.kube.lan that originate within the cluster to instead resolve to internal cluster ip addresses, and not 192.168.8.222

Both are the responsibility of standard kubernetes DNS add-on.

i want queries for the rest of the universe to continue to leave the cluster

You need to use the forward plugin for that.

networkop on Sep 22, 2022