docker-ipv6nat: Exits with "unable to detect hairpin mode (is the docker daemon running?)"

Hi all,

With Docker 20.10.6 the ipv6nat function is fully intergrated (experimental). You can add the following flags to your daemon.json: { "ipv6": true, "fixed-cidr-v6": "fd00::/80", "experimental": true, "ip6tables": true }

To so to be 100% clear, this “backport” in the RedHat packages happened between versions 1.8.4-15.el8 and 1.8.4-15.el8_3.3?

Correct, see the changelog here: https://centos.pkgs.org/8/centos-baseos-x86_64/iptables-services-1.8.4-15.el8_3.3.x86_64.rpm.html

I spoke to Phil Sutter from RedHat, who did both the upstream patch as well as its backport into RHEL8.3.

The commit in question is here. To quote Phil:

Sadly it is not part of an official release yet, ETA is v1.8.7.

About the issue we’re seeing in the Docker container:

Basically it’s a problem with data representation inside the container. The iptables binary in there doesn’t respect the reduced payload expression length and due to absence of the (not needed) bitwise expression assumes the full address is being matched.

So aside from the workaround (downgrading as detailed here) I guess the only solution would be to either wait for 1.8.7 (and its Alpine edge packages) or build a patched version and ship that in the container image.

Well that fixed it. 🤕

Wow, that was quite the journey. Great you figured it out! And thanks for the detailed fix.

Let’s leave it at this. I’ll keep an eye out for more reports of this issue.

Makes me wonder if I could have mounted the host xtables-nft-multi binary in the container to fix it. Probably only if it was statically linked, since the container runs on Alpine (MUSL based IIRC).

Yeah, that’s usually a no-go, for that exact reason.

Just pushed out a new release v0.4.4 which contains the fix for this issue! Docker images for all architectures are on Docker Hub as :0.4.4 and :latest. Thanks everyone!

If this is the issue, couldn’t it be fixed by upgrading the ipv6net Docker container to use a newer version of iptables? Maybe as an opt-in (eg. a new tag)

That seems to be the plan, once iptables is updated in Alpine. We all seem to agree that we should wait until it is “stable” before doing that.

I can confirm that this error occurs with a fresh install of CentOS. The downgrade solution helped for me.

Well that fixed it. 🤕

Feel free to close this issue if you want, since it seems to be a problem with an external package. Though if you think it is a compatibility issue, I would be happy to help you continue to debug it (though not on my home prod system).

Full fix detailed, in case anyone else has the exact same stuck package scenario:

# Get all "old" packages
$ wget http://mirror.centos.org/centos/8/BaseOS/x86_64/os/Packages/iptables-1.8.4-15.el8.x86_64.rpm
$ wget http://mirror.centos.org/centos/8/BaseOS/x86_64/os/Packages/iptables-ebtables-1.8.4-15.el8.x86_64.rpm
$ wget http://mirror.centos.org/centos/8/BaseOS/x86_64/os/Packages/iptables-libs-1.8.4-15.el8.x86_64.rpm
$ wget http://mirror.centos.org/centos/8/BaseOS/x86_64/os/Packages/iptables-services-1.8.4-15.el8.x86_64.rpm

$ sudo yum downgrade ./iptables-*

# Destroy the container, just in case any loaded kernel modules stick around
$ docker rm -f ipv6nat

# Rebooting was the only thing that fixed it for me
$ sudo reboot

# Recreate the container
$ docker run -d --name ipv6nat --cap-drop ALL --cap-add NET_ADMIN --cap-add NET_RAW --network host --restart unless-stopped -v /var/run/docker.sock:/var/run/docker.sock:ro robbertkl/ipv6nat
# or whatever your command is

Thanks for all your help tracking down what was causing the problem!

Your system uses nftables and I think it’s only iptables-nft in the container that talks directly to nftables. I don’t think it’s even using the iptables installed on the host.

You are right I think. I guess somehow the new package version has some bug that interacts with the kernel incorrectly, and then saves (and then later prints) the rules incorrectly?? Yeah, doesn’t make sense to me either. I wouldn’t even know how to go about reporting this as a bug to the package maintainer. I guess I would need to prove that the rule actually got saved wrong somehow.

Also wondering how iptables on the host could have an effect in the first place: I don’t think it’s even used?

Makes me wonder if I could have mounted the host xtables-nft-multi binary in the container to fix it. Probably only if it was statically linked, since the container runs on Alpine (MUSL based IIRC).