podman: podman doesn't start `aardvark-dns` when in an LXC container

Issue Description

Inside an LXC container, podman does not start aardvark-dns. This breaks container DNS resolution when a podman network is used because podman still properly re-writes the containers /etc/resolve.conf to point to (expected to be running) aardvark-dns.

I do not see this issue when using a very similar environment, but on metal rather than an LXC.

After starting the aardvark-dns daemon by hand with /usr/libexec/podman/aardvark-dns --config /run/containers/storage/networks/aardvark-dns -p 53 run, everything works as expected for the remainder of the LXC container’s lifetime.

I understand that this environment may not be available to the podman developers. I am willing to do some debugging, but so far I have not been able to identity any error or other logging which may indicate the issue.

Steps to reproduce the issue

Start (in my case) a Fedora 38 LXC container.
podman network create test
podman run --rm -it --network test fedora:38 bash
getent hosts example.org <-- this returns nothing.

Without the --network flag getent hosts example.org returns 2606:2800:220:1:248:1893:25c8:1946 example.org.

After starting aardvark-dns with /usr/libexec/podman/aardvark-dns --config /run/containers/storage/networks/aardvark-dns -p 53 run on the host, then the DNS resolution works even with the --network flag on podman.

Describe the results you received

DNS resolution doesn’t work.

Describe the results you expected

DNS resolution to work.

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

Fedora 38


Client:       Podman Engine
Version:      4.5.0
API Version:  4.5.0
Go Version:   go1.20.2
Built:        Fri Apr 14 15:42:22 2023
OS/Arch:      linux/amd64

I can not easily upgrade the environment to podman 4.5.1, but based on the changelog, I do not think this issue is addressed.



### Podman in a container

Yes

### Privileged Or Rootless

Privileged

### Upstream Latest Release

No

### Additional environment details

Within an LXC container.

### Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

About this issue

Original URL
State: open
Created a year ago
Reactions: 1
Comments: 22 (11 by maintainers)

Most upvoted comments

I just wanted to report that this issue is not stale. We noticed it again today after some system updates.

abalmos on Oct 6, 2023

I’m not sure if it is the same issue, but I have hit upon aardvark-dns not starting in an LXC container as well and perhaps it will help. It happens as:

Create an unprivileged LXC container, so that it is running in a user namespace
Login to the container as a non-root user
sudo podman network create test
sudo podman run --rm -it --network test --log-level debug fedora:38 bash

The issue here is that since podman is running in a user namespace (the LXC container), it considers it as rootless. netavark in rootless mode uses systemd-run --user to start aardvark-dns. That however requires a proper systemd user session to be created… sudo does not create such a session, so the systemd-run fails:

[DEBUG netavark::dns::aardvark] Spawning aardvark server
[DEBUG netavark::dns::aardvark] start aardvark-dns: ["systemd-run", "-q", "--scope", "--user", "/usr/libexec/podman/aardvark-dns", "--config", "/run/containers/storage/networks/aardvark-dns", "-p", "53", "run"]
Failed to connect to bus: No medium found

It works when you login as root directly and do not use sudo/su to switch privileges. It could be the same issue if Quadlet does not run podman with a proper user session, which is possible, I guess.

aither64 on Jun 19, 2023