kops: Worker nodes fail to join cluster due to broken gossip DNS (AWS / 1.26)

/kind bug

1. What kops version are you running? The command kops version, will display this information.

1.26.0-beta.1

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

1.26.0

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

kops create cluster \
    --name=k8s-aws-83e2.k8s.local \
    --node-count 3 \
    --master-size c5.large \
    --node-size m5.2xlarge \
    --cloud aws \
    --cloud-labels "Provisioner"="kops","installation_name"="k8s-aws-83e2" \
    --zones us-east-2a --ssh-public-key public_key.pub \
    --state=s3://k8s-aws-kops/k8s-aws-83e2 \
    --kubernetes-version 1.26.0 \
    --yes

And then:

kops validate cluster \
    --name=k8s-aws-83e2.k8s.local \
    --state s3://k8s-aws-kops/k8s-aws-83e2

5. What happened after the commands executed?

Validation failed due to worker nodes not joining the cluster within timeout:

Validating cluster k8s-aws-83e2.k8s.local

INSTANCE GROUPS
NAME				ROLE		MACHINETYPE	MIN	MAX	SUBNETS
control-plane-us-east-2a	ControlPlane	c5.large	1	1	us-east-2a
nodes-us-east-2a		Node		m5.2xlarge	3	3	us-east-2a

NODE STATUS
NAME			ROLE		READY
i-0182cfe92957781dc	control-plane	True

VALIDATION ERRORS
KIND	NAME						MESSAGE
Machine	i-01e9bdf0184a2aa1a				machine "i-01e9bdf0184a2aa1a" has not yet joined cluster
Machine	i-0363e370402e946a8				machine "i-0363e370402e946a8" has not yet joined cluster
Machine	i-0f91d8686618d5cac				machine "i-0f91d8686618d5cac" has not yet joined cluster
Pod	kube-system/coredns-865c5db65f-jfxvd		system-cluster-critical pod "coredns-865c5db65f-jfxvd" is pending
Pod	kube-system/coredns-autoscaler-7f4ddcc6c7-5dksr	system-cluster-critical pod "coredns-autoscaler-7f4ddcc6c7-5dksr" is pending

Validation Failed
Error: validation failed: cluster not yet healthy

6. What did you expect to happen?

Worker nodes to successfully join the cluster.

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2022-12-26T16:59:19Z"
  name: k8s-aws-83e2.k8s.local
spec:
  api:
    loadBalancer:
      class: Network
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudConfig:
    awsEBSCSIDriver:
      enabled: true
      version: v1.14.0
    manageStorageClasses: true
  cloudControllerManager:
    allocateNodeCIDRs: true
    clusterCIDR: 100.64.0.0/10
    clusterName: k8s-aws-83e2.k8s.local
    configureCloudRoutes: false
    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0
    leaderElection:
      leaderElect: true
  cloudLabels:
    Provisioner: kops
    installation_name: k8s-aws-83e2
  cloudProvider: aws
  clusterDNSDomain: cluster.local
  configBase: s3://k8s-aws-kops/k8s-aws-83e2/k8s-aws-83e2.k8s.local
  configStore: s3://k8s-aws-kops/k8s-aws-83e2/k8s-aws-83e2.k8s.local
  containerRuntime: containerd
  containerd:
    logLevel: info
    runc:
      version: 1.1.4
    version: 1.6.14
  docker:
    skipInstall: true
  etcdClusters:
  - backups:
      backupStore: s3://k8s-aws-kops/k8s-aws-83e2/k8s-aws-83e2.k8s.local/backups/etcd/main
    cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2a
      name: a
    memoryRequest: 100Mi
    name: main
    version: 3.5.6
  - backups:
      backupStore: s3://k8s-aws-kops/k8s-aws-83e2/k8s-aws-83e2.k8s.local/backups/etcd/events
    cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-2a
      name: a
    memoryRequest: 100Mi
    name: events
    version: 3.5.6
  externalDns:
    provider: dns-controller
  iam:
    allowContainerRegistry: true
    legacy: false
  keyStore: s3://k8s-aws-kops/k8s-aws-83e2/k8s-aws-83e2.k8s.local/pki
  kubeAPIServer:
    allowPrivileged: true
    anonymousAuth: false
    apiAudiences:
    - kubernetes.svc.default
    apiServerCount: 1
    authorizationMode: Node,RBAC
    bindAddress: 0.0.0.0
    cloudProvider: external
    enableAdmissionPlugins:
    - NamespaceLifecycle
    - LimitRanger
    - ServiceAccount
    - DefaultStorageClass
    - DefaultTolerationSeconds
    - MutatingAdmissionWebhook
    - ValidatingAdmissionWebhook
    - NodeRestriction
    - ResourceQuota
    etcdServers:
    - https://127.0.0.1:4001
    etcdServersOverrides:
    - /events#https://127.0.0.1:4002
    featureGates:
      CSIMigrationAWS: "true"
      InTreePluginAWSUnregister: "true"
    image: registry.k8s.io/kube-apiserver:v1.26.0@sha256:d230a0b88a3daf14e4cce03b906b992c8153f37da878677f434b1af8c4e8cc75
    kubeletPreferredAddressTypes:
    - InternalIP
    - Hostname
    - ExternalIP
    logLevel: 2
    requestheaderAllowedNames:
    - aggregator
    requestheaderExtraHeaderPrefixes:
    - X-Remote-Extra-
    requestheaderGroupHeaders:
    - X-Remote-Group
    requestheaderUsernameHeaders:
    - X-Remote-User
    securePort: 443
    serviceAccountIssuer: https://api.internal.k8s-aws-83e2.k8s.local
    serviceAccountJWKSURI: https://api.internal.k8s-aws-83e2.k8s.local/openid/v1/jwks
    serviceClusterIPRange: 100.64.0.0/13
    storageBackend: etcd3
  kubeControllerManager:
    allocateNodeCIDRs: true
    attachDetachReconcileSyncPeriod: 1m0s
    cloudProvider: external
    clusterCIDR: 100.96.0.0/11
    clusterName: k8s-aws-83e2.k8s.local
    configureCloudRoutes: false
    featureGates:
      CSIMigrationAWS: "true"
      InTreePluginAWSUnregister: "true"
    image: registry.k8s.io/kube-controller-manager:v1.26.0@sha256:26e260b50ec46bd1da7352565cb8b34b6dd2cb006cebbd2f35170d50935fb9ec
    leaderElection:
      leaderElect: true
    logLevel: 2
    useServiceAccountCredentials: true
  kubeDNS:
    cacheMaxConcurrent: 150
    cacheMaxSize: 1000
    cpuRequest: 100m
    domain: cluster.local
    memoryLimit: 170Mi
    memoryRequest: 70Mi
    nodeLocalDNS:
      cpuRequest: 25m
      enabled: false
      image: registry.k8s.io/dns/k8s-dns-node-cache:1.22.15
      memoryRequest: 5Mi
    provider: CoreDNS
    serverIP: 100.64.0.10
  kubeProxy:
    clusterCIDR: 100.96.0.0/11
    cpuRequest: 100m
    enabled: false
    image: registry.k8s.io/kube-proxy:v1.26.0@sha256:1e9bbe429e4e2b2ad32681c91deb98a334f1bf4135137df5f84f9d03689060fe
    logLevel: 2
  kubeScheduler:
    featureGates:
      CSIMigrationAWS: "true"
      InTreePluginAWSUnregister: "true"
    image: registry.k8s.io/kube-scheduler:v1.26.0@sha256:34a142549f94312b41d4a6cd98e7fddabff484767a199333acb7503bf46d7410
    leaderElection:
      leaderElect: true
    logLevel: 2
  kubelet:
    anonymousAuth: false
    cgroupDriver: systemd
    cgroupRoot: /
    cloudProvider: external
    clusterDNS: 100.64.0.10
    clusterDomain: cluster.local
    enableDebuggingHandlers: true
    evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
    featureGates:
      CSIMigrationAWS: "true"
      InTreePluginAWSUnregister: "true"
    kubeconfigPath: /var/lib/kubelet/kubeconfig
    logLevel: 2
    podInfraContainerImage: registry.k8s.io/pause:3.6@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db
    podManifestPath: /etc/kubernetes/manifests
    protectKernelDefaults: true
    registerSchedulable: true
    shutdownGracePeriod: 30s
    shutdownGracePeriodCriticalPods: 10s
  kubernetesApiAccess:
  - 0.0.0.0/0
  - ::/0
  kubernetesVersion: 1.26.0
  masterKubelet:
    anonymousAuth: false
    cgroupDriver: systemd
    cgroupRoot: /
    cloudProvider: external
    clusterDNS: 100.64.0.10
    clusterDomain: cluster.local
    enableDebuggingHandlers: true
    evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
    featureGates:
      CSIMigrationAWS: "true"
      InTreePluginAWSUnregister: "true"
    kubeconfigPath: /var/lib/kubelet/kubeconfig
    logLevel: 2
    podInfraContainerImage: registry.k8s.io/pause:3.6@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db
    podManifestPath: /etc/kubernetes/manifests
    protectKernelDefaults: true
    registerSchedulable: true
    shutdownGracePeriod: 30s
    shutdownGracePeriodCriticalPods: 10s
  networkCIDR: 172.20.0.0/16
  networking:
    cilium:
      agentPrometheusPort: 9090
      bpfCTGlobalAnyMax: 262144
      bpfCTGlobalTCPMax: 524288
      bpfLBAlgorithm: random
      bpfLBMaglevTableSize: "16381"
      bpfLBMapMax: 65536
      bpfNATGlobalMax: 524288
      bpfNeighGlobalMax: 524288
      bpfPolicyMapMax: 16384
      clusterName: default
      cpuRequest: 25m
      disableCNPStatusUpdates: true
      disableMasquerade: false
      enableBPFMasquerade: false
      enableEndpointHealthChecking: true
      enableL7Proxy: true
      enableNodePort: true
      enableRemoteNodeIdentity: true
      hubble:
        enabled: false
      identityAllocationMode: crd
      identityChangeGracePeriod: 5s
      ipam: kubernetes
      memoryRequest: 128Mi
      monitorAggregation: medium
      sidecarIstioProxyImage: cilium/istio_proxy
      toFqdnsDnsRejectResponseCode: refused
      tunnel: vxlan
      version: v1.12.5
  nonMasqueradeCIDR: 100.64.0.0/10
  podCIDR: 100.96.0.0/11
  secretStore: s3://k8s-aws-kops/k8s-aws-83e2/k8s-aws-83e2.k8s.local/secrets
  serviceClusterIPRange: 100.64.0.0/13
  sshAccess:
  - 0.0.0.0/0
  - ::/0
  subnets:
  - cidr: 172.20.32.0/19
    name: us-east-2a
    type: Public
    zone: us-east-2a
  topology:
    dns:
      type: Private
    masters: public
    nodes: public

---

 apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2022-12-26T16:59:19Z"
  name: control-plane-us-east-2a
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20221206
  instanceMetadata:
    httpPutResponseHopLimit: 3
    httpTokens: required
  machineType: c5.large
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-east-2a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2022-12-26T16:59:19Z"
  name: nodes-us-east-2a
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20221206
  instanceMetadata:
    httpPutResponseHopLimit: 1
    httpTokens: required
  machineType: m5.2xlarge
  maxSize: 3
  minSize: 3
  role: Node
  subnets:
  - us-east-2a

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

Troubleshooting steps on one the worker nodes:

$ journalctl -u kops-configuration
- Logs begin at Mon 2022-12-26 17:00:00 UTC, end at Wed 2022-12-28 13:01:50 UTC. --
Dec 26 17:00:17 i-0f91d8686618d5cac systemd[1]: Starting Run kOps bootstrap (nodeup)...
Dec 26 17:00:17 i-0f91d8686618d5cac nodeup[1491]: nodeup version 1.26.0-beta.1 (git-v1.26.0-beta.1)
Dec 26 17:00:17 i-0f91d8686618d5cac nodeup[1491]: W1226 17:00:17.126504    1491 main.go:133] got error running nodeup (will retry in 30s): failed to get node config from server: lookup kops-controller.internal.k8s-aws-83e2.k8s.local on 127.0.0.53:53: server misbehaving

(the above DNS resolution failure repeats forever)

$ journalctl -u protokube
-- Logs begin at Mon 2022-12-26 17:00:00 UTC, end at Wed 2022-12-28 13:03:50 UTC. --
-- No entries --

$ systemctl status protokube
Unit protokube.service could not be found.

$ nslookup kops-controller.internal.k8s-aws-83e2.k8s.local
Server:         127.0.0.53
Address:        127.0.0.53#53

** server can't find kops-controller.internal.k8s-aws-83e2.k8s.local: SERVFAIL

$ cat /etc/resolv.conf
# <comments reducted>

nameserver 127.0.0.53
options edns0 trust-ad
search us-east-2.compute.internal

ubuntu@i-0f91d8686618d5cac:~$ cat /etc/hosts
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Notice there isn’t a kops-managed entry for kops-controller.internal.k8s-aws-83e2.k8s.local.

On the master node, however, there’s a corresponding entry in /etc/hosts:

# Begin host entries managed by kops - do not edit
172.20.59.12    api.internal.k8s-aws-83e2.k8s.local kops-controller.internal.k8s-aws-83e2.k8s.local
# End host entries managed by kops

For comparison, the exact same kops create command successfully creates a cluster with kops v1.25.1. On a 1.25.1 worker node, I can see that /etc/hosts does have the missing entry:

# Begin host entries managed by kops - do not edit
172.20.52.193   api.internal.k8s-aws-e7a6.k8s.local kops-controller.internal.k8s-aws-e7a6.k8s.local
# End host entries managed by kops