cri-o: Ubuntu 18.04: DNS resolution broken after scheduling hostport enabled pods

Description

Cri-O breaks DNS resolution once pod with “Hostport” is scheduled. I’m logging this issue here, as i was unable to reproduce this issue on Docker 17.03 and Containerd 1.1.3/1.2.0-beta.

The symptoms are similar to https://github.com/kubernetes/kubeadm/issues/273 .

Steps to reproduce the issue:

Schedule a pod with hostPort container spec configured.
Kubernetes shows the pod in ImagePullBack error.
DNS resolution is not working anymore. However kubelet and crio services later disabled.

Additional information you deem important (e.g. issue happens only occasionally):

Output of crio --version:

cri-o-1.11 Xenial Package

Additional environment details (AWS, VirtualBox, physical, etc.):

Hetzner Cloud and Scaleway
OS: Ubuntu 18.04 Bionic
Kubernetes 1.11.2
CNI 0.7.1
CNI runtime: Issue observed using Canal, Cillium and Weave

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 2
Comments: 15 (7 by maintainers)

Most upvoted comments

@runcom, @Bregor The usual way to tacle the issue with containerd and docker is to enforce the path to the proper resolv.conf in the kubelet.

This can be acheived using via the command line argument --resolv-conf=/run/systemd/resolve/resolv.conf or via Kubelet configuration file.

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/etc/kubernetes/ssl/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.3.0.10"
podCIDR: "10.2.0.0/16"
runtimeRequestTimeout: "15m"
tlsCertFile: "/etc/kubernetes/ssl/node.pem"
tlsPrivateKeyFile: "/etc/kubernetes/ssl/node-key.pem"
staticPodPath: "/etc/kubernetes/manifests"
failSwapOn: false
resolvConf: /run/systemd/resolve/resolv.conf

But it does not work with cri-o.

fjudith on Sep 4, 2018

@runcom I did not try with containerd, but docker-18.06 “broken” just like cri-o

Bregor on Sep 3, 2018