moby: Docker swarm load balancing not working over private network

Managed to fix it for me.

Since requests to the webserver in my swarm are handed over by the ingress lb but responses are timing out i started testing if it has something to do with the size of the packets which are transferred. While playing with ping -s $packetsize i realized that packets bigger (if i remember correct) 1420 bytes are dropped.

Wireguard uses a MTU of 1420 and Docker uses a MTU of 1500 per default.

After lowering the MTU of the Ingress Overlay Network everything is working as expected for me.

All steps are done on a manager node:

1. Stop all running services in your swarm
2. docker network rm ingress (remove the ingress overlay network since we are creating a new one with a lower mtu)
3. Create a new ingress overlay network with your prefered subnet & gateway

docker network create --driver overlay --ingress --opt com.docker.network.driver.mtu=1280 --subnet 10.11.0.0/24 --gateway 10.11.0.1 ingress 

1. systemctl restart docker (don’t know if i was not patient enough but that populated the new created network to all other nodes)
2. bring back your services

At this point everything was working for me. I’m pretty sure a MTU of 1280 is a bit too low in this case but im still testing.

Hope i could help some other people facing the same problems.

grettings

Another important thing that I’ve noticed!

While recreating the ingress with the correct MTU fixes Outside World -> Swarm Cluster communication, it does not fix communication within your cluster!

I have a app that routes traffic between four of my containers, and I have two Swarm Nodes

• Node 1: http-coordinator, container1, container2
• Node 2: container3, container4

I was having issue that, sometimes, requests to the container3 and container4 were timing out, while requests to container1 and container2 weren’t.

19:12:36.710 [eventLoopGroupProxy-4-1] WARN  n.p.l.c.m.i.InteractionsHttpCoordinator - Something went wrong while trying to forward the request!
io.ktor.client.plugins.HttpRequestTimeoutException: Request timeout has expired [url=http://cinnamon-production-2:12212, request_timeout=unknown ms]
        at io.ktor.client.engine.cio.EndpointKt$setupTimeout$timeoutJob$1.invokeSuspend(Endpoint.kt:247)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:749)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)

And the issue is… drum roll

To fix this, you need to set the MTU in your docker-compose.yml files too!

networks:
  default:
    driver: overlay
    driver_opts:
      com.docker.network.driver.mtu: 1450

Replace 1450 with your network interface’s MTU! :3

Source

OK I have changed my wireguard MTU to 1500 which is the MTU of docker and my openstack interface, it’s working too. So I hope it won’t split the packets now I have the same MTU everywhere 😃

Yes try Wireguard, really simple, just a little time consuming to administrate. As a DevOps, I like when it’s automated like swarm 😄. Maybe if you have issues with wireguard, I coud help you like you did for me 👍

Just a question about MTU, Could I setup the value to another like 1300 or 1370 ? I always saw value like 1280 or 1400 on forum, there is a specific rule to calculate MTU ? Then, could we change the wireguard MTU 1420 to 1500 ? Already tested but it doesnt work on my case…

Haha, right when you were typing your comment, I was updating my previous comment with newer things that I found while playing around with the MTU size. :3

The issue is that Docker uses MTU 1500 by default and it doesn’t inherit the MTU set by your network interface.

The MTU that you are setting in the ingress network should be the same MTU set in your network interface!

To get the MTU of your network, use ip a and find the interface that you have set as the advertise-addr parameter (if you haven’t set it, it is the default network interface)

3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether ...

In this case, we can see that the MTU of my network is 1450, so we are going to set our ingress network to also be 1450!

So, check what’s the MTU that Wireguard is using by using ip a and checking the mtu xxxx field, then use that value when creating your ingress network. It should, hopefully, work fine without any issues. 😃

(At least on my side it worked fine after I bumped from 1400 to 1450. My network interface has its MTU set to 1450 because it is a VXLAN network)

Ok with your setup it’s working !! 🥇 thanks for all. My multi cloud swarm over wireguard is working well now 😃

@bdoublet91 I’m back from my vacation, so I decided to play around a bit more:

I created a Swarm with sudo docker swarm init --default-addr-pool 192.168.128.0/18 --advertise-addr 172.29.10.1

After creation, I inspected the overlay network to check how it looks like before we start to mess around with it.

swarm@docker-swarm-manager-1:~$ sudo docker network ls
NETWORK ID     NAME              DRIVER    SCOPE
a54488e50cb3   bridge            bridge    local
5c9af67afb9a   docker_gwbridge   bridge    local
1cdb51e53a49   host              host      local
ll9lfov7eu7i   ingress           overlay   swarm
90ec800e99d5   none              null      local
6ec10d2e1af0   swarm_default     bridge    local
swarm@docker-swarm-manager-1:~$ sudo docker inspect ingress
[
    {
        "Name": "ingress",
        "Id": "ll9lfov7eu7i6wodq5xsrc0yn",
        "Created": "2022-08-07T17:04:21.685399396Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.128.0/24",
                    "Gateway": "192.168.128.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "034a4390f781d05b88b8b6b9c52d35a9b407a0b90b4c58cb9c171cdb637ff0c2",
                "MacAddress": "02:42:c0:a8:80:02",
                "IPv4Address": "192.168.128.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "108faab843d1",
                "IP": "172.29.10.1"
            },
            {
                "Name": "078912a29389",
                "IP": "172.29.11.1"
            }
        ]
    }
]

I added a worker node to my swarm and deployed my personal website with two replicas to my Swarm, so both VMs (manager and worker) had one container running my website.

Tried curl’ing my website from outside of my swarm VM…

root@doge-reborn:~# curl 10.29.10.1:40000/br/
^C <--- it hung, so I exited curl
root@doge-reborn:~# curl 10.29.10.1:40000/br/
<!DOCTYPE html>
<html lang="pt">
  <head>
    <meta charset="utf-8">
root@doge-reborn:~# curl 10.29.10.1:40000/br/
^C <--- once again, rip
root@doge-reborn:~# curl 10.29.10.1:40000/br/
^C <--- *picture of a sad cat here*
root@doge-reborn:~# curl 10.29.10.1:40000/br/
<!DOCTYPE html>
<html lang="pt">
  <head>
...

So as we can see, it isn’t working correctly: Some requests work fine, some requests hang indefinitely…

Docker, by default, uses 1500 MTU for ingress communication, it doesn’t inherit the MTU configuration of the network interface. So, what we need to do is change the MTU of the ingress to match the network interface that we are using for intra node communication!

To get the MTU of your network, use ip a and find the interface that you have set as the advertise-addr parameter (if you haven’t set it, it is the default network interface)

3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether ...

In this case, we can see that the MTU of my network is 1450, so we are going to set our ingress network to also be 1450!

First, let’s inspect the ingress network again

swarm@docker-swarm-manager-1:~$ sudo docker inspect ingress
[
    {
        "Name": "ingress",
        "Id": "ll9lfov7eu7i6wodq5xsrc0yn",
        "Created": "2022-08-07T17:04:21.685399396Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.128.0/24",
                    "Gateway": "192.168.128.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "421fbe812ee7194cee388c5f2869024b2b9a9fa8ba7ee9c2f84d6d13eb85c9be": {
                "Name": "powercms_powercms.2.lrn7pwe6wyw3rmzdrw6mxmf3l",
                "EndpointID": "247336a13f52f6d0fc26996e7c9dbb6f875a3df0c444a4c70fa80c2d6754e510",
                "MacAddress": "02:42:c0:a8:80:06",
                "IPv4Address": "192.168.128.6/24",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "034a4390f781d05b88b8b6b9c52d35a9b407a0b90b4c58cb9c171cdb637ff0c2",
                "MacAddress": "02:42:c0:a8:80:02",
                "IPv4Address": "192.168.128.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "108faab843d1",
                "IP": "172.29.10.1"
            },
            {
                "Name": "078912a29389",
                "IP": "172.29.11.1"
            }
        ]
    }
]

There are new things here, but we can safely ignore them.

First, we need to remove all running stacks, then…

• docker network rm ingress
• (You may need to restart Docker with systemctl restart docker)
• docker network create --driver overlay --ingress --opt com.docker.network.driver.mtu=1450 --subnet 192.168.128.0/24 --gateway 192.168.128.1 ingress
  • The subnet and gateway are on the IPAM section!
• Restart Docker with systemctl restart docker
• Restart Docker on every node

Now let’s inspect the network again

swarm@docker-swarm-manager-1:~$ sudo docker inspect ingress
[
    {
        "Name": "ingress",
        "Id": "n4kc7x5bsilk13w7r8zlqcxmk",
        "Created": "2022-08-07T17:16:43.164266372Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.128.0/24",
                    "Gateway": "192.168.128.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "905eb24af09900820bb24b3ca831b16e60a2c154dea519a01e11785247229e06",
                "MacAddress": "02:42:c0:a8:80:02",
                "IPv4Address": "192.168.128.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.mtu": "1400",
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "37c1d2b7ef95",
                "IP": "172.29.10.1"
            },
            {
                "Name": "93cf234fada7",
                "IP": "172.29.11.1"
            }
        ]
    }
]

The configuration should be similar to your previous configuration. Don’t worry if the com.docker.network.driver.overlay.vxlanid_list changed, that’s used for VXLAN tagging. (And you can set it to 4096 by using --opt com.docker.network.driver.overlay.vxlanid_list=4096 when creating the network)

And then everything should work fine! I tried flooding curl 10.29.10.1:40000/br/ and none of the requests failed.

@bdoublet91 let’s suppose you have a Linux network interface that has the MTU set to 1450 (it is a VXLAN connection). The VXLAN IP range is 172.16.0.0/12, and you want your containers in Docker Swarm to use the 192.168.128.0/18 range:

docker swarm init --default-addr-pool 192.168.128.0/18

Then

docker network rm ingress
docker network create --driver overlay --ingress --opt com.docker.network.driver.mtu=1280 --subnet 192.168.128.0/18 --gateway 192.168.128.1 ingress
systemctl restart docker

Sadly I didn’t have too much time to play with it to check if it really fixed the issue or not, because one day after I was experimenting with Swarm after getting too burnt out with Kubernetes, I went on vacation, but on my limited testing (using curl to spam my personal website), which has two replicas, hosted on two Docker Swarm VMs).

Before the change some connections were randomly hanging up when trying to query the website, and after the fix the issue went away, so ymmv!

Technically the MTU should also work fine if it was set to 1400 instead of 1280, this is something that I still need to play around with after I’m back from vacation.

Yes, the issue still persists for current wireguard (0.0.20180910-wg1) and docker-ce (18.06.1-ce).

I have 2 nodes, both are are active and reachable over internal addresses but every 2nd request to docker service fails.

Sadly, I stuck at the same point. Could not figure out what blocks requests between docker nodes.

Not really, I’ve found a job. 😅

Gonna try to reproduce the issue today and report back.