origin: 503-errors when re-deploying router in openshift 3.11 and 3.09

When deploying the default haproxy router in Openshift 3.11 (and 3.09) it throws 503 errors:

2019-06-20T06:45:44+00:00 10.128.6.1 haproxy[18]: Proxy azurehealthcheck started.
2019-06-20T06:45:44+00:00 10.128.6.1 haproxy[18]: Proxy public started.
2019-06-20T06:45:44+00:00 10.128.6.1 haproxy[18]: Proxy public_ssl started.
2019-06-20T06:45:44+00:00 10.128.6.1 haproxy[18]: Proxy be_sni started.
2019-06-20T06:45:44+00:00 10.128.6.1 haproxy[18]: Proxy fe_sni started.
2019-06-20T06:45:44+00:00 10.128.6.1 haproxy[18]: Proxy be_no_sni started.
2019-06-20T06:45:44+00:00 10.128.6.1 haproxy[18]: Proxy fe_no_sni started.
2019-06-20T06:45:44+00:00 10.128.6.1 haproxy[18]: Proxy openshift_default started.
2019-06-20T06:45:44+00:00 10.128.6.1 haproxy[19]: 127.0.0.1:33384 [20/Jun/2019:06:45:44.643] public openshift_default/<NOSRV> 1/-1/-1/-1/0 503 3278 - - SC-- 2/1/0/0/0 0/0 "HEAD / HTTP/1.1"
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[27]: Proxy azurehealthcheck started.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[27]: Proxy public started.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[27]: Proxy public_ssl started.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[27]: Proxy be_sni started.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[27]: Proxy fe_sni started.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[27]: Proxy be_no_sni started.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[27]: Proxy fe_no_sni started.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[27]: Proxy openshift_default started.
[...]
[more proxies started]
[...]
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Stopping frontend public in 0 ms.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Stopping frontend public_ssl in 0 ms.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Stopping backend be_sni in 0 ms.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Stopping frontend fe_sni in 0 ms.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Stopping backend be_no_sni in 0 ms.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Stopping frontend fe_no_sni in 0 ms.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Stopping backend openshift_default in 0 ms.
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Proxy azurehealthcheck stopped (FE: 1 conns, BE: 0 conns).
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Proxy public stopped (FE: 1 conns, BE: 0 conns).
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Proxy public_ssl stopped (FE: 0 conns, BE: 0 conns).
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Proxy be_sni stopped (FE: 0 conns, BE: 0 conns).
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Proxy fe_sni stopped (FE: 0 conns, BE: 0 conns).
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Proxy be_no_sni stopped (FE: 0 conns, BE: 0 conns).
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Proxy fe_no_sni stopped (FE: 0 conns, BE: 0 conns).
2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[19]: Proxy openshift_default stopped (FE: 0 conns, BE: 1 conns).

2019-06-20T06:45:45+00:00 10.128.6.1 haproxy[28]: 127.0.0.1:33392 [20/Jun/2019:06:45:45.061] public openshift_default/<NOSRV> 1/-1/-1/-1/0 503 3278 - - SC-- 1/1/0/0/0 0/0 "HEAD / HTTP/1.1"

Version

openshift v3.11.0+cbab8ee-94
kubernetes v1.11.0+d4cacc0

Steps To Reproduce

1. Redirect the output of the default router to rsyslog, as described here: https://github.com/PhilipGough/rsyslog-container
2. Follow the logs of the default router
3. deploy the default router
4. notice the errors

Current Result

503 errors in haproxy when deploying the router

Expected Result

No errors. 😃

Additional Information

We noticed this with prometheus monitoring the routers on 500 errors: