podman: DNS does not work in containers if host uses local server

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

  1. Be in a network that prohibits external DNS queries, disable external DNS communication or just use some only-locally available hostname in step 3.

  2. Setup local DNS server/forwarder (e.g. systemd-resolved) so that the local address is in /etc/resolv.conf

  3. Start any container (without --network host) and try to resolve a hostname (e.g. podman run --rm -it fedora curl -v ifconfig.me)

Describe the results you received: curl: (6) Could not resolve host: ifconfig.me

Describe the results you expected: No error (some IP address)

Additional information you deem important (e.g. issue happens only occasionally): The contents of /etc/resolv.conf are:

search virt
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844
nameserver 10.0.2.3
options edns0

Which would normally work (although I might not want to send my DNS requests somewhere else because I might have services available in a local network), but I am in a network that prohibits external DNS queries, so that doesn’t work.

If I leave just the slirp4netns nameserver there (echo nameserver 10.0.2.3 >/etc/resolv.conf) it works in a VM where I am trying to reproduce this issue. However on my original host, where I discovered this, 10.0.2.3 is still inaccessible (even though the version and the command-line of slirp4netns is identical, apart from the PID argument).

Output of podman version:

Version:            1.3.1
RemoteAPI Version:  1
Go Version:         go1.12.2
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.2
  podman version: 1.3.1
host:
  BuildahVersion: 1.8.2
  Conmon:
    package: podman-1.3.1-1.git7210727.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: c9a4c48d1bff85033b7fc9b62d25961dd5048689'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 2884521984
  MemTotal: 4133556224
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 644870144
  SwapTotal: 644870144
  arch: amd64
  cpus: 4
  hostname: fedora30.virt
  kernel: 5.0.9-301.fc30.x86_64
  os: linux
  rootless: true
  uptime: 19m 40.53s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/nert/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/nert/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /tmp/1000
  VolumePath: /home/nert/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.): I am trying this in a Fedora 30 VM, clean install, as that is the easiest and cleanest reproducer I can get. I cannot reproduce the issue related to my local environment in there.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 32 (16 by maintainers)

Commits related to this issue

Most upvoted comments

so on fedora (as the main developer platform for podman people) and also rhel-8 systemd-resolved is the default. so using a properly configured systemd-resolved on host can’t be used as a name server for containers run by root!? and the only solution is to use --net host!

why did you close this issue???

A /etc/resolv.conf generated by systemd-resolved looks like this:

nameserver 127.0.0.53
options edns0
search some.dom

As a result podman seems to remove the nameserver line and adds the “upstream” DNS servers directly. Bypassing systemd-resolved on the host may work in some scenarios, but it breaks others. Consider a corporate VPN connections where “upstream” is not defined. (We have at least two “upstreams” when connected)

Bypassing the host’s systemd-resolved has at least the following problems:

  • Upstream/Internet DNS servers do not know about corporate specific DNS entries
  • Corporate DNS servers may deliver different addresses than public DNS servers
  • Corporate DNS servers may not deliver results for public stuff
  • Corporate DNS servers may change, so unable to use fixed IP addresses

So whatever which DNS server I choose by setting --dns=<ip>, I will never get the same results as talking to the systemd-resolved running on the host.

How to fix that? Well, I guess it’s not possible at all. podman would have to replace nameserver 127.0.0.53 with something that is forwarded or hosted on the host. But systemd-resolved is listening on the loopback interface only and does not allow (AFAIK) to change / configure that.

[Edit] The combination of systemd-resolved and podman is the default for Fedora users. So privileged containers are quite unusable when corporate VPNs are in the game.

@rhatdan Can this issue be reopened?

From what I read above podman is still unreliable when there are non-trivial dns setups on the host.

Ah, I see. systemd-resolvd has apparently decided that the rules for everyone else don’t apply to it, and is doing its own thing. So that’s fun.

Docker has the same issue.

Although I think some new features have been added to allow users to share the hosts localhost network. At least in rootless mode.

@mheon @giuseppe WDYT?