moby: Slow networking inside containers (tried host and bridge)

I have two containers on the same host (RHEL 7.2, 3.10.0-514.6.1.el7.x86_64 SMP, dm-thinpool storage) that I’m networking together. Network transfers between the two containers take twice as long compared to a setup with the client code running natively outside of Docker on the same host.

Container A (based on ubuntu:latest) is a server which listens on port 47301. I start it like this:

docker run -d --restart=unless-stopped -p 47301:47301 -p 9000:9000 -v /somedir:/somedir:ro --name container-a container-a:latest

Container B connects to the server in Container A using SSL. I start it like this:

docker run -d --restart=unless-stopped --network=host --name container-b container-b:latest

The code in Container B is set to connect to myhostname.fqdn:47301. (I’ve also tested with localhost:47301, no difference.) Container B is based on Debian Jessie.

The “host” network shows container B is connected (although, curiously, I don’t see any mention of container A):

> docker network inspect host
[
    {
        "Name": "host",
        "Id": "adablahblah",
        "Created": "2017-02-01T09:18:21.348731359-05:00",
        "Scope": "local",
        "Driver": "host",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "743blahblahblah": {
                "Name": "container-b",
                "EndpointID": "af6yadayadayada",
                "MacAddress": "",
                "IPv4Address": "",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

I also tried creating my own bridge network:

> docker network create --driver bridge myproject

Then, I specified --network=myproject when starting both container A and container B. In that scenario, the code in container-B is set to connect to container-a:47301. This works fine, but there is no improvement in performance. The network transfers still take about twice as long when compared to the container-B code running natively on the same host (outside of Docker).

My Docker daemon does not have IPv6 enabled.

Output of docker version:

Client:
 Version:      1.13.0
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:55:28 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:55:28 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 3
 Running: 3
 Paused: 0
 Stopped: 0
Images: 181
Server Version: 1.13.0
Storage Driver: devicemapper
 Pool Name: os-dkthinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 15.35 GB
 Data Space Total: 51.64 GB
 Data Space Available: 36.29 GB
 Metadata Space Used: 4.583 MB
 Metadata Space Total: 25.17 MB
 Metadata Space Available: 20.58 MB
 Thin Pool Minimum Free Space: 5.163 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-514.6.1.el7.x86_64
Operating System: Scientific Linux 7.2 (Nitrogen)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.686 GiB
Name: myhostname
ID: stuff
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Can anyone help? I’d expect a ~10% decrease in networking performance using bridge/bridge container networking. We could live with that, but ~100% decrease in performance is unacceptable for our project.

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Reactions: 1
  • Comments: 39 (7 by maintainers)

Most upvoted comments

@esabol Did you find the issue? I think I’m hitting the same problem.

From what I can tell from your configuration here, you are using an exposed port on the host to communicate back into the container. This is most definitely going to be much slower because all local traffic must go through a proxy process, this is due to limitations in linux bridge networking. One way to get around this and continue using it as you are is to set --userland-proxy=false on the daemon which enables hairpin bridge mode… the reason this is not the default is because of known kernel issues which cause major system degradation… I would not recommend doing this.

Really I would have the containers communicate directly rather than going through this port forward. The easiest way to do this is to create a network:

$ docker network create myApp

And then have each container use this network:

$ docker run --net=myApp <other options>

Then each container can communicate with the other via container name.

Sharing for others benefit. I used docker network create and overhead of docker networking was not that bad (Ubuntu 20.10, kernel 5.8, Docker 19.03), but when adding --subnet option to the network, with no other changes network performance suffered greatly. 9 minutes vs 2 hours for some network testing task (testssl.sh).

EDIT: Tried with --security-opt apparmor=unconfined and upgraded to docker-ce 20.10.3, issue persists with subnet.

No. I’ve been meaning to go back and re-test/re-benchmark everything after having upgraded a bunch of things, but haven’t gotten around to it, unfortunately.

My solution was not to run my worker/client inside a Docker container (container B above), which isn’t a solution, but it resulted in the networking performance we needed.

@maksymilian-palka I forgot to mention this here, but I also saw a increase in throughput performance when using the --privileged flag on container creation. Apparently accessing host devices (e.g. the network interfaces for receiving and sending packets) is slowed down by some kind of capability check. The --privileged flag should remove those capability checks. I plan to release a paper containing more information about this issue soon. I will link it when it’s done. In the mean time can you try to confirm my results by using the privileged flag in your setup?