portainer: Reccuring error in my logs "unexpected status code | status_code=403"

Yep, someone beat you to the root cause in #11596 so I raised it internally, and the team have a fix in place for our next release. The fix reduces the check frequency to once every 24 hours (except when hard refreshing in the browser), which should mean simply using Portainer should no longer push the rate limit up.

Once the release including this is out we’ll loop back to this (and the other) thread to confirm this is fixed, and if so we’ll close the issue(s) off.

I can confirm that this problem is still present in Portainer 2.20.1 😢.

However, I finally found the cause of the issue after modifying Portainer to log the URL in addition to the status code.

2024/04/25 10:03PM ERR github.com/portainer/portainer/api/http/client/client.go:92 > unexpected status code | status_code=403 url=https://api.github.com/repos/portainer/portainer/releases/latest
2024/04/25 10:03PM ERR github.com/portainer/portainer/api/http/client/client.go:92 > unexpected status code | status_code=403 url=https://api.github.com/repos/portainer/portainer/releases/latest
2024/04/25 10:03PM ERR github.com/portainer/portainer/api/http/client/client.go:92 > unexpected status code | status_code=403 url=https://api.github.com/repos/portainer/portainer/releases/latest

As it can be seen there, the URL that causes the sporadic 403 errors is:

https://api.github.com/repos/portainer/portainer/releases/latest

This URL is called by Portainer itself to check if there are newer versions available in GitHub.

Why does it return 403? Because Portainer is calling this URL very frequently, and GitHub is applying rate limiting. If you open a browser with that URL after seeing those 403 errors, you will get the error from GitHub:

{"message":"API rate limit exceeded for X.X.X.X. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}

Portainer seems to be calling this URL when you switch windows or tabs in the browser, as you can see this error much more frequently when you do so. This is because the Portainer UI triggers a version check everytime the window/tab regains focus (not a good idea in my opinion and should be considered a bug).

The reason this error does not appear always is because it depends on the state of rate-limiting from Github for your current public IP address. If you leave Portainer for a bit (enough for the rate limiting to be lifted) you will not see these errors until the rate limit is triggered again.

@jamescarppe (or anyone in Portainer Team) does the above help fixing this quite long standing issue? 😃

EDIT: The frequency of calling GitHub API should be fixed, but perhaps it also would be a good idea to add a command-line flag to disable new version checks entirely if desired?

However, I finally found the cause of the issue (…)

Wow, this is impressive @hhromic - it is not common to see such good bug reports. I will have to get better at digging down

@johackim

I accidentally reproduced your issue in my cluster. I am trying to connect to an environment via an agent and am receiving context deadline exceeded toasts in the UI when this happens:

2022/11/23 02:06AM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403
2022/11/23 02:06AM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403
2022/11/23 02:07AM ERR github.com/portainer/portainer/api@v0.0.0-20221011053128-b0938875dc5c/http/client/client.go:94 > unexpected status code | status_code=403

In my case, I had an environment connected to a Portainer server via an Agent. I tried connecting that same environment to a different Portainer server.

Thanks!