portainer: [BE] Unable to retrieve networks, stacks, versions, etc.
Bug description After Updating Portainer-EE to V2.17.1 and the Portainer_agent on my Synology NAS to 2.17.1, the Errors “Unable to retrieve stacks”, “Unable to retrieve networks”, “Unable to retrieve version”… pop up while accessing the Environment from the control panel.
Expected behavior The Environment should be accessible without errors and the Containers hosted within Docker should show up.
Portainer Logs portainer_agent on Synology NAS: https://pastebin.com/LMFM5hEB portainer-ee on VPS: https://pastebin.com/dWYPBkDR
Steps to reproduce the issue:
tbh I don’t know -_-
- Update portainer-ee and portainer_agent(on Synology NAS) to V2.17.1
- Login on the Portainer control panel
- Click on the Environment of the NAS
- See error
Technical details:
- Portainer version: V2.17.1
- Docker version (managed by Portainer): VPS: 20.10.23 - NAS: 20.10.3
- Kubernetes version (managed by Portainer): -
- Platform (windows/linux): VPS: Ubuntu 20.04 - NAS: DSM 7.1.1-42962 Update 4
- Command used to start Portainer (
docker run -p 9443:9443 portainer/portainer
): Regular Docker Commands as found in the Docs(except the volumes Binding on the NAS. That is customized) - Browser: Safari, Opera(different machine)
- Use Case (delete as appropriate): Using Portainer at Home AND Using Portainer in a Commercial setup.
- Have you reviewed our technical documentation and knowledge base? Yes
Additional context The connection to the portainer_agent is handled over a Cloudflare tunnel(in order to not expose any Ports of my Home network) but accessing the hostname:port in a Browser outputs the regular API stating “{“message”:“Missing request signature headers”,“details”:“Unauthorized”}”
Edit: The Agent on a different VPS(Same Version as NAS) works completely fine…
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 18
@LPwithPaul Try to disable the “Brotli” in your Cloudflare domain dashboard, under speed/optimization section. Works for me.
It seems this “http: proxy error: invalid character” error is caused by accidentally reading compressed data from resp.Body and io.Copy() the data to the downstream as uncompressed data.
I set up a local http reverse proxy to dump requests between the agent and portioner. All response jsons are good. And they are all compressed. After I intensionally remove the “accept-encoding” header to disable compression. The problem is gone.
I’m not good at Go, so I can’t dig into it further.
@tamarahenson
Update:
I’m sure this issue is caused by the Brotli compressed response. The Portainer forwards browser header, which “accept-encoding” contains “br”. If the remote responses with Brotli compressed data (Agent itself don’t support it, but a reverse proxy may do). Then
https://github.com/portainer/portainer/blob/36541093328115a4cf54522852afe76fc9434de4/api/http/proxy/factory/utils/response.go#L100
https://github.com/portainer/portainer/blob/36541093328115a4cf54522852afe76fc9434de4/api/http/proxy/factory/utils/json.go#L41
Json decoder recevied raw Brotli compressed data.