docker-flow-proxy: unpredictable 503 Service Unavailable with multiple instances of DFP

Description

I have DFP deployed as Docker Swarm stack 2 replicas across 2 different nodes. In front of DFP there is a loadbalancer. At the moment LB points to just one node in order to reduce the debugging surface. For the sake of the test I have also passed LB and executed requests directly against DFP and found out it does not have impact.

Steps to reproduce the issue:

1. Deploy DFP … wait
2. Deploy application stack on application node 1
3. Wait, to give cluster the chance to not have excuses
4. Repetitively execute requests against DFP
5. Deploy another version of stack on application node 2
6. Wait, to give cluster the chance to not have excuses
7. Repetitively execute requests against DFP

Describe the results you received:

Usually when first app stack is deployed DFP does not return 503. But almost always it does randomly when second stack is deployed.

I have also checked /cfg/haproxy.cfg on both proxy nodes but they are the same. I have also tried DO_NOT_RESOLVE_ADDR=true but without effect.

Once I decrease the number of replicas to 1, everything is stable.

In the proxy log on the node where the request reaches I get something like this:

2018/06/09 15:43:11 HAPRoxy: 10.255.0.9:43070 [09/Jun/2018:15:43:11.898] services~ services/<NOSRV> -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET /console/ HTTP/1.1"

… although the service (the container is up & listening, the entry is in haproxy.cfg) is up.

Describe the results you expected:

I expect no 503 for replicas > 1 as well.

Additional information you deem important (e.g. issue happens only occasionally):

Additional environment details (AWS, VirtualBox, physical, etc.):

DFP tested: 18.05.26-51 and previously also 17.09.22-18 Docker engine: 17.09.0-ce