moby: Docker networking fails after iptables service is restarted
Docker networking fails permanently on all new containers after restarting the iptables service. Steps to reproduce on CentOS 7.1 (some output elided for brevity):
$ sudo yum -y install docker iptables-services
$ sudo systemctl start docker
$ sudo docker run --rm centos bash -c "ping www.docker.com"
PING www.docker.com (162.242.195.82) 56(84) bytes of data.
64 bytes from docker.com (162.242.195.82): icmp_seq=1 ttl=61 time=114 ms
$ sudo systemctl restart iptables
$ sudo docker run --rm centos bash -c "ping www.docker.com"
ping: unknown host www.docker.com
A workaround to restore networking to containers is to restart the Docker daemon:
$ sudo systemctl restart docker
$ sudo docker run --rm centos bash -c "ping www.docker.com"
PING www.docker.com (162.242.195.82) 56(84) bytes of data.
64 bytes from docker.com (162.242.195.82): icmp_seq=1 ttl=61 time=114 ms
What should be fixed in Docker so that reloading iptables rules won’t break its networking? This is a major issue because it breaks networking for all containers on the host.
About this issue
- Original URL
- State: open
- Created 9 years ago
- Reactions: 25
- Comments: 40 (9 by maintainers)
Definitely having a command like “docker network reload-firewall” would be nice so that if you run into a situation you can fix the firewall without having to restart all of your containers…
What about adding a command or mechanism that invoke that network setup phase (updating iptables) without restarting docker daemon?
iptables-save is a messy non-solution. Any semi-complex set of rules needs to be written in a more structured way, with comments. You also do not want to save Docker’s temporary rules since the saved rules file is usually used on system startup (especially when it results from an unclean shutdown without a save and the saved rules file is a little older), when half of them are obsolete because the containers they belonged to no longer exist.
Docker simply needs a command to recreate its rules without restarting docker itself. It does not have to determine automatically when that command has to be called, it is easy enough to add it to any script flushing and recreating firewall rules. Honestly, I can’t see how anyone can run Docker on a production server with any sort of availability requirement without a way to use iptables for other things.
it’s been six years. How hard is it to add a docker command to re-apply docker’s firewall rules? 😕
I’m surprised that nobody has mentioned it, but there is actually a built-in, working solution to this problem: the Docker “live restore” option (which has been around since 2016!): https://docs.docker.com/config/containers/live-restore/
After configuring that option, you need to restart docker once to load it, and after that, the containers will continue to run during all subsequent restarts (and not only will the restart process be very fast, but it will also restore the firewall rules if they don’t exist).
I’m using it in multiple Ansible & Docker environments, and it works like a charm.
This is a major flaw of Docker that the team refuses to fix. There must be a way for Docker to reload the containers’ iptables rules.
The fact that this ticket is still open is one of the things I regularly point to when people ask why I consider Docker far from production ready (the others include the lack of proper garbage collection).
Quite frankly the whole Docker container hype is a bit of a bad joke while tickets like this one are completely ignored.
This helped me a lot with this issue: https://unrouted.io/2017/08/15/docker-firewall/ Baiscally doing things with iptables-restore -n to avoid flushing existing rules… Docker rules untouched.
I want to recreate iptables rules without restart dockerd
Maybe we should just run Docker inside of Docker, so that we don’t have to keep the ephemeral nature of these rules from causing problems? \s
libvirtd handles SIGHUP for flushed firewall rules: https://libvirt.org/firewall.html Why can’t Docker handle this?
I have a slightly different use case and have been frustrated there isn’t a workaround. I’m pushing out very specific iptables rules using ansible and those remove the docker created rules. So I wrote some shell scripts (leveraging netfilter-persistent) that make a backup of the running rules, allow me to push my ansible rules, then restore only the docker specific ones. Its a bit of a hack but gets me closer to my goal. If these might be useful for you then you can pull them down at:
https://github.com/paklids/docker-iptables-restore
Has anyone using firewalld tried setting their default zone to the docker zone, and then applying their public rules to that zone?
The docs say that anything in the DOCKER-USER chain will get applied before any other rules. Which made me think that the solution to this issue is to put all our rules into DOCKER-USER.
My thought is that the docker zone might do that for firewalld.
Anyway, as soon as I have time, I’ll be experimenting to see if that approach works. I’m just kinda hoping someone else might have already done so. 😃
Merry Christmas, btw.
we are also having this problem in GitLab CI runners here: when the firewall system (
ferm) reloads because of outside conditions, it flushes the firewall rules and breaks Docker. it would be nice ifservice docker reloadwould also reload the firewall rules (it seems we needservice docker restartnow) so that we could have a PropagateReloadsTo relationship between the two services.as things stand now, we had to add this horrible override in
/etc/systemd/system/ferm.service.d/override.conf:i understand the docker people might not feel responsible for this, but maybe this small tweak could be sufficient to solve this problem?
alternatively, we’re also considering using podman as a container runner as well. we suspect it might be better suited for short-lived jobs like this anyways, and will likely not suffer from this problem.
You can not save and restore Docker iptables rules externally unless you have only static, long-running Docker containers with exposed ports.
What Docker really just needs is the same thing most other, similar software does in this case. Call a configurable script on each event (startup, container start, container stop, shutdown) and pass all the relevant data to the script as parameters.