portainer: http: proxy error: dial tcp 127.0.0.1:50480: connect: connection refused

Description I have one Portainer instance running and two Portainer edge agents on different virtual machines. It was working but since a couple of weeks I have an issue and the Portainer instance can not connect to the edge agents any more.

Trying to connect to an agent, I get the following error: Failed loading endpoint Endpoint is unreachable and there is no snapshot available for offline browsing.

Heartbeat says “good”.

Portaineredge agent log: 2021/06/21 17:51:25 [WARN] [os,options] [message: the CAP_HOST_MANAGEMENT environment variable is deprecated and will likely be removed in a future version of Portainer agent] 2021/06/21 17:51:25 [INFO] [main] [message: Agent running on Docker platform] 2021/06/21 17:51:25 [INFO] [main,edge] [message: Edge key loaded from options] 2021/06/21 17:51:25 [INFO] [http] [server_addr: 172.17.0.2] [server_port: 9001] [secured: false] [api_version: 2.5.1] [message: Starting Agent API server] 2021/06/21 17:52:00 client: Connecting to ws://portainer.tld:8000

On the Portainer instance, I get:

2021-06-21 17:53:19	stderr	2021/06/21 17:53:19 http: proxy error: dial tcp 127.0.0.1:51018: connect: connection refused
2021-06-21 17:53:15	stderr	2021/06/21 17:53:15 [DEBUG] [chisel,monitoring] [endpoint_id: 28] [status: ACTIVE] [status_time_seconds: 4.976697] [message: endpoint tunnel monitoring]
2021-06-21 17:52:06	stderr	2021/06/21 17:52:06 http: proxy error: dial tcp 127.0.0.1:58696: connect: connection refused
2021-06-21 17:52:05	stderr	2021/06/21 17:52:05 [DEBUG] [chisel,monitoring] [endpoint_id: 28] [status: ACTIVE] [status_time_seconds: 4.985799] [message: endpoint tunnel monitoring]
2021-06-21 17:42:23	stderr	2021/06/21 17:42:23 http: proxy error: dial tcp 127.0.0.1:61208: connect: connection refused
2021-06-21 17:42:15	stderr	2021/06/21 17:42:15 [DEBUG] [chisel,monitoring] [endpoint_id: 28] [status: ACTIVE] [status_time_seconds: 0.376469] [message: endpoint tunnel monitoring]
2021-06-21 17:38:41	stderr	2021/06/21 17:38:41 http: proxy error: dial tcp 127.0.0.1:50480: connect: connection refused
2021-06-21 17:38:35	stderr	2021/06/21 17:38:35 [DEBUG] [chisel,monitoring] [endpoint_id: 28] [status: ACTIVE] [status_time_seconds: 0.375208] [message: endpoint tunnel monitoring]
2021-06-21 17:37:49	stderr	2021/06/21 17:37:49 http: proxy error: dial tcp 127.0.0.1:60759: connect: connection refused
2021-06-21 17:37:45	stderr	2021/06/21 17:37:45 [DEBUG] [chisel,monitoring] [endpoint_id: 28] [status: ACTIVE] [status_time_seconds: 5.383998] [message: endpoint tunnel monitoring]
2021-06-21 17:36:26	stderr	2021/06/21 17:36:26 http: proxy error: dial tcp 127.0.0.1:58064: connect: connection refused
2021-06-21 17:36:25	stderr	2021/06/21 17:36:25 [DEBUG] [chisel,monitoring] [endpoint_id: 27] [status: ACTIVE] [status_time_seconds: 7.708604] [message: endpoint tunnel monitoring]
2021-06-21 17:35:59	stderr	2021/06/21 17:35:59 http: proxy error: dial tcp 127.0.0.1:57070: connect: connection refused
2021-06-21 17:35:55	stderr	2021/06/21 17:35:55 [DEBUG] [chisel,monitoring] [endpoint_id: 28] [status: ACTIVE] [status_time_seconds: 5.191327] [message: endpoint tunnel monitoring]

I am confused why Portainer is trying to connect to 127.0.0.1. Shouldn’t it be the IP of the edge agent?

Steps to reproduce the issue:

1. Start portainer
2. Add endpoint
3. Start portainer edge agent
4. Try access the endpoint over portainer

Technical details:

• Portainer version: 2.5.1
• Target Docker version (the host/cluster you manage): 20.10.3 and 20.10.7
• Platform (windows/linux): linux
• Command used to start Portainer (docker run -p 9000:9000 portainer/portainer): Just standard
• Browser: All