moby: iptables failed - No chain/target/match by that name

Bug Report Info

docker version: Client version: 1.7.1 Client API version: 1.19 Go version (client): go1.4.2 Git commit (Client): 786b29d OS/Arch (client): linux/amd64 Server version: 1.7.1 Server API verson: 1.19 Go version (server): go1.4.2 Git commit (server): 786b29d OS/Arch (server): linux/amd64

docker info: Containers: 41 Images: 172 Storage Driver: devicemapper Pool Name: docker-253:2-4026535945-pool Pool Blocksize: 65.54 kB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 7.748 GB Data Space Total: 107.4 GB Data Space Available: 99.63 GB Metadata Space Used: 12.55 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.135 GB Udev Sync Supported: true Deferred Removal Enabled: true Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.93-RHEL7 (2015-01-28) Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 3.10.0-123.el7.x86_64 Operating SYstem: CentOS Linux 7 (Core) CPUs: 24 Total Memory: 125.6 GiB Name: <hostname> ID: <id>

uname -a: Linux <hostname> 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.): Physical iptables version 1.4.21

How reproducible: Random

Steps to Reproduce:

  1. Start container with exposed ports mapped to host ports
  2. Stop container
  3. Repeat, good luck.

Actual Results:

Cannot start container <container id>: iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.23 --dport 4000 -J ACCEPT: iptables: No chain/
target/match by that name.

Expected Results:

Container starts without a problem.

Additional info:

I’ll also mention these containers are being launched via Apache Mesos (0.23.0) using Marathon. Appears similar to #13914.

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 55
  • Comments: 63 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Exactly the same issue here as @shayts7 is describing. Workaround for now is to restart the daemon:

service docker restart

I have met a similar problem and it was solved by running this command: # iptables -t filter -N DOCKER Hope it helps!

It happened to us as well, but in our case iptables -t filter -L -v -n showed that DOCKER chain exists, only when checking the nat table using iptables -t nat -L -v -n we found out that somehow DOCKER chain was disappear…


Chain PREROUTING (policy ACCEPT 6402K packets, 388M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 981K packets, 62M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 1001K packets, 63M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 514K packets, 31M bytes)
 pkts bytes target     prot opt in     out     source               destination
  83M 5047M FLANNEL    all  --  *      *       192.168.0.0/16       0.0.0.0/0
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.135       192.168.18.135       tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.167       192.168.18.167       tcp dpt:8081
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.167       192.168.18.167       tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.172       192.168.18.172       tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.186       192.168.18.186       tcp dpt:8081
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.186       192.168.18.186       tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.194       192.168.18.194       tcp dpt:53
    0     0 MASQUERADE  udp  --  *      *       192.168.18.194       192.168.18.194       udp dpt:53
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.197       192.168.18.197       tcp dpt:3000
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.198       192.168.18.198       tcp dpt:1936
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.198       192.168.18.198       tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.198       192.168.18.198       tcp dpt:88
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.198       192.168.18.198       tcp dpt:80
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.2         192.168.18.2         tcp dpt:53
    0     0 MASQUERADE  udp  --  *      *       192.168.18.2         192.168.18.2         udp dpt:53
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.4         192.168.18.4         tcp dpt:1936
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.4         192.168.18.4         tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.4         192.168.18.4         tcp dpt:88
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.4         192.168.18.4         tcp dpt:80
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.5         192.168.18.5         tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.6         192.168.18.6         tcp dpt:3000
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.8         192.168.18.8         tcp dpt:8081
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.8         192.168.18.8         tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.9         192.168.18.9         tcp dpt:8081
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.9         192.168.18.9         tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.10        192.168.18.10        tcp dpt:8080

Chain FLANNEL (1 references)
 pkts bytes target     prot opt in     out     source               destination
5481K  332M ACCEPT     all  --  *      *       0.0.0.0/0            192.168.0.0/16
 426K   27M MASQUERADE  all  --  *      *       0.0.0.0/0           !224.0.0.0/4

After restarting docker daemon everything worked fine and we could see DOCKER chain came back to nat table:

Chain PREROUTING (policy ACCEPT 5765 packets, 347K bytes)
 pkts bytes target     prot opt in     out     source               destination
 1592 96542 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 1236 packets, 75057 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 3135 packets, 203K bytes)
 pkts bytes target     prot opt in     out     source               destination
    1    77 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 2423 packets, 159K bytes)
 pkts bytes target     prot opt in     out     source               destination
  83M 5047M FLANNEL    all  --  *      *       192.168.0.0/16       0.0.0.0/0
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.135       192.168.18.135       tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.167       192.168.18.167       tcp dpt:8081
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.167       192.168.18.167       tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.172       192.168.18.172       tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.186       192.168.18.186       tcp dpt:8081
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.186       192.168.18.186       tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.194       192.168.18.194       tcp dpt:53
    0     0 MASQUERADE  udp  --  *      *       192.168.18.194       192.168.18.194       udp dpt:53
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.197       192.168.18.197       tcp dpt:3000
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.198       192.168.18.198       tcp dpt:1936
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.198       192.168.18.198       tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.198       192.168.18.198       tcp dpt:88
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.198       192.168.18.198       tcp dpt:80
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.2         192.168.18.2         tcp dpt:53
    0     0 MASQUERADE  udp  --  *      *       192.168.18.2         192.168.18.2         udp dpt:53
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.4         192.168.18.4         tcp dpt:1936
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.4         192.168.18.4         tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.4         192.168.18.4         tcp dpt:88
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.4         192.168.18.4         tcp dpt:80
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.5         192.168.18.5         tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.6         192.168.18.6         tcp dpt:3000
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.8         192.168.18.8         tcp dpt:8081
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.8         192.168.18.8         tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.9         192.168.18.9         tcp dpt:8081
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.9         192.168.18.9         tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.10        192.168.18.10        tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.2         192.168.18.2         tcp dpt:53
    0     0 MASQUERADE  udp  --  *      *       192.168.18.2         192.168.18.2         udp dpt:53
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.5         192.168.18.5         tcp dpt:3000
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.6         192.168.18.6         tcp dpt:5601
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.7         192.168.18.7         tcp dpt:8201
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.7         192.168.18.7         tcp dpt:8200
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.8         192.168.18.8         tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.9         192.168.18.9         tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.10        192.168.18.10        tcp dpt:8081
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.10        192.168.18.10        tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.11        192.168.18.11        tcp dpt:8081
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.11        192.168.18.11        tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.12        192.168.18.12        tcp dpt:1936
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.12        192.168.18.12        tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.12        192.168.18.12        tcp dpt:88
    0     0 MASQUERADE  tcp  --  *      *       192.168.18.12        192.168.18.12        tcp dpt:80

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 to:192.168.18.2:53
    0     0 DNAT       udp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 to:192.168.18.2:53
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:3210 to:192.168.18.5:3000
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5601 to:192.168.18.6:5601
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8201 to:192.168.18.7:8201
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8200 to:192.168.18.7:8200
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8050 to:192.168.18.8:8080
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9002 to:192.168.18.9:8080
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8041 to:192.168.18.10:8081
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8040 to:192.168.18.10:8080
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8081 to:192.168.18.11:8081
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8080 to:192.168.18.11:8080
   27  1620 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:1936 to:192.168.18.12:1936
    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:192.168.18.12:443
  139  8340 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:88 to:192.168.18.12:88
   24  1440 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:192.168.18.12:80

Chain FLANNEL (1 references)
 pkts bytes target     prot opt in     out     source               destination
5489K  332M ACCEPT     all  --  *      *       0.0.0.0/0            192.168.0.0/16
 427K   27M MASQUERADE  all  --  *      *       0.0.0.0/0           !224.0.0.0/4

If someone has a clue for why the chain disappear I’ll be more than happy to hear about it.

Hello everyone,

I’m using coreos and have this problem too but only on my master.

Running iptables -t nat -N DOCKER solves the problem, pods are automatically created and everything is fine. I’m looking to know why this chain is removed on my master and not on my workers.

follow @fredrikaverpil .Thank you.

I try:

ip link delete docker0

systemctl restart docker

this issue occurs when I restart container after I stop the firewalld

docker version: Docker version 1.9.1, build a34a1d5 docker info: uname -a: Linux databus0 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:

  1. docker run -d --name=sth -p4444:4444 sometth
  2. killall firewalld
  3. docker retart sth

Describe the results you received:

Error response from daemon: Cannot restart container sth: failed to create endpoint sth on network bridge: iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 4444 -j DNAT --to-destination 172.17.0.5:4444 ! -i docker0: iptables: No chain/target/match by that name.
 (exit status 1)
Error: failed to restart containers: [sth]

Describe the results you expected: restart ok

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

Hi All,

I faced the same problem, it fixed for me.

Enter below command, it will clear all chains.

iptables -t filter -F

iptables -t filter -X

Then restart Docker Service using below comamnd

systemctl restart docker

I hope it will work.

This worked for me !! on all CentOS7.2 systems

ip link delete docker0
systemctl restart docker

@veuncent docker creates the docker chain in IPTables rules on startup; if some other system (such as firewalld) is removing those rules after docker is started, this error can occur. Make sure the docker daemon i started after firewalld.

Hello everyone!

I have faced this issue, and found out that after running my firewall script it removes the DOCKER chain, reason why it gets this error… so, when restarting docker service, it will fix this problem, because docker recreate the chains used by its service.

To fix:

service docker stop
service docker start

But, it would be nice if when running any create container command docker check if there is its chain or recreate that.

Would it be possible to update it?

Sorry not being possible to contribute with a pull request.

Was having this issue. For us it turned out docker was starting before our firewall persistence (iptables-persistent) and its rules were getting overwritten. I resolved by removing the package as we were using it for only 1 rule.

There are ways to keep it working side by side by either ensuring docker starts after iptables-persistent(https://groups.google.com/forum/#!topic/docker-dev/4SfOwCOmw-E) or by adding whatever rules the docker service adds into the persistent iptables configuration(didn’t test this). May be of help @Seraf, @shayts7

This is not a docker bug but maybe it should be addressed in docs or something

This seems to only happen on CentOS 7 for me.

This is what I did

stop firewalld

sudo systemctl stop firewalld
sudo systemctl disable firewalld

Restart your machine

sudo reboot

As long as you’ve put –restart=always to your docker instance. When your machine is reboot, the docker instance should be running, and the port should be binded. I believe this issue is specificly to CentOS7 family who uses firewalld instead of iptables.

I have solved the issue by typing service iptables restart and service docker restart . Hope it helps.

Hi There. Im runing a VM INFO:

   Static hostname: n/a
Transient hostname: aIP-OF-MY-MACHINE
         Icon name: computer-vm
           Chassis: vm
        Machine ID: d4047bd0916d41d38b6b97ff7b5f2b3d
           Boot ID: 61456d6912e24569985f0e9343bd8179
    Virtualization: qemu
  Operating System: openSUSE Tumbleweed
       CPE OS Name: cpe:/o:opensuse:tumbleweed:20200817
            Kernel: Linux 5.8.0-1-default
      Architecture: x86-64

Docker Version:

Client:
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        48a66213fe17
 Built:             Mon Aug  3 00:00:00 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       48a66213fe17
  Built:            Mon Aug  3 00:00:00 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.1.5_catatonit
  GitCommit:

So, Im working arround a almost 1 week to solve this issue! My MAIN issue is i have detected some random disconects to my VPS, disconects are afected on all ports lossing all acess! I made some research and i find on ```/var/log/firewalld logs the issues that I will mention below OUTPUT:

...
2020-09-15 01:21:23 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
2020-09-15 01:21:23 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-09-15 01:21:26 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
...

I already have executed this commands :

iptables -t filter -F
iptables -t filter -X

Then restart Docker Service using below comamnd

ip link delete docker0
systemctl restart docker

I have tried to make some this commands, and deinstalled docker to remove dockers configs… without much sucess… 👎 …

It is sad that this is happening! I have some work to do in a production environment

firewalled remove DOCKER’s rule do something that systemctl restart docker to solves

Hi

I used docker-compose command to start elasticSerach, logstash and Kibana and run normally for several hours, then ELK can not work properly. So I tried to restart the docker elasticSearch or logstatsh or Kibana but met similar problem.

Steps to reproduce the issue:

  1. Use docker-compose to start ELK docker
  2. Observed data loading stopped in Kibana dashboard
  3. Try to restart docker elasticSearch or logstash or Kibana, then met the results mentioned below.
  4. If run sudo service docker restart command, then can start ELK again

Describe the results you received: Error response from daemon: Cannot restart container <container ID>: driver failed programming external connectivity on endpoint dockerelk_elasticsearch_1 : (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 9300 -j DNAT --to-destination 172.18.0.2:9300 ! -i: iptables: No chain/target/match by that name.

Describe the results you expected: Docker can run normally without this problem and no need restart.

Additional information you deem important (e.g. issue happens only occasionally): The problem happened after several hours normal running.

Output of docker version:

Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:07:28 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:07:28 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 8
 Running: 0
 Paused: 0
 Stopped: 8
Images: 5
Server Version: 17.03.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 77
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
init version: 949e6fa
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.866 GiB

Additional environment details (AWS, VirtualBox, physical, etc.): uname -a Linux scav-dev.fordme.com 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux

Try creating the chain in iptables by running iptables -N DOCKER

and if that doesn’t work, try upgrading docker and iptables