portainer: Unable to log in to 2.30.0 STS

Before you start please confirm the following.

Problem Description

Unable to log in to 2.30.0 STS as per #12664 again! This should have been fixed, how has it regressed again and got through CI tests?!

Expected Behavior

To be able to log in and modify stacks!

Actual Behavior

Can’t log in without clearing cookies each time.

Image

When logged in, I cannot do anything useful, such as redeploying stacks or logging out. Error is once again a 403 error complaining about invalid origin! See below.

Image

Image

Steps to Reproduce

  1. Log in
  2. Can’t - error appears in popup
  3. Clear cookies
  4. Logs in successfully
  5. Attempt redeploy existing stack - fails with error about not being able to find compose.yml

Portainer logs or screenshots

Image

Portainer version

2.30.0

Portainer Edition

Business Edition (BE/EE) with 5NF / 3NF license

Platform and Version

Docker 28.1.1

OS and Architecture

Raspbian OS - ARM64

Browser

Chrome + Safari

What command did you use to deploy Portainer?

services:
  portainer:
    container_name: portainer-ee
    image: portainer/portainer-ee:sts
    ports: # Ports published to host
      - 9443:9443 # Web UI
    environment:
      - VIRTUAL_HOST=portainer.iot
      - VIRTUAL_PORT=9443
    volumes:
      - data:/data # Volume storing settings
      - /var/run/docker.sock:/var/run/docker.sock # Bind mount of host's Docker socket
    restart: unless-stopped

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 6
  • Comments: 110 (25 by maintainers)

Most upvoted comments

I have got around this issue by adding the following to my nginx reverse proxy, seems to be all working again now proxy_set_header Origin "";

Same problem here.

Fixed it by modifying nginx proxy:

location / {
        proxy_set_header Origin "";
        proxy_set_header Referer "portainer:9000";
        proxy_pass http://portainer:9000;
    }

@pmharris77 Yes, thank you for the suggestion. We did consider using TRUSTED_ORIGINS as one of the solutions. However, we ultimately decided against it because it would impact all users, not just those running Portainer behind a reverse proxy. Users would have to consult the documentation and determine the correct values to set, which isn’t ideal, especially for those unfamiliar with these concepts.

Overall, it felt like a less user-friendly approach. Instead, we chose a solution that preserves the existing experience for most users without requiring configuration changes.

Before releasing the feature, our internal team tested the following scenarios:

  • HTTPS only
  • HTTP only
  • HTTPS-to-HTTP
  • HTTPS-to-HTTPS

So that’s the full context. If it turns out that no other workaround solves the remaining edge cases, we may revisit the idea. But our goal remains the same, to keep the Portainer experience as seamless and hassle-free as possible

Hi, why this bug being released into Community Edition after reported for months, can you explain that? We are expecting this works as original.

Mention @oscarzhou-portainer here, this issue already hits CE for 3 days, and we should expect a possible fix now I think.

For those on the LTS release, please try the following https://github.com/portainer/portainer/issues/12748 We have released Portainer 2.27.9 LTS which contains the fix for this, and will be releasing this to STS soon.

@Mister-Hope Thank you for sharing the detailed configuration. It’s extremely helpful. While the team is working on the issue, a quick workaround for your setup would be to apply the following changes:

proxy_set_header Host $host;
proxy_set_header Origin ""; # remove this line

with help of chatgpt i solved this problem:

Fix for “Forbidden - origin invalid” in Portainer 2.30.0 using nginx

After upgrading to Portainer 2.30.0, I encountered 403 errors when trying to perform actions like removing images or logging out. This turned out to be caused by stricter CSRF checks in Portainer that validate the Origin and X-Forwarded-Proto headers.

Fix

In my nginx reverse proxy config, I fixed the issue by adding the following two lines inside the location / block:

proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Origin "";

Same problem here on: Server Version: 2.27.7 LTS

Same issue here, 403 is returned to /api/auth. Deleting the cookie works to sign in. Sign out fails once more. I’ve set it up behind a reverse proxy and it’s OK as per the guide.

@Nick-Portainer @oscarzhou-portainer just downloaded and tested with the latest lts version and the trusted domain flag and it is working again

Same problem here.

I’m using Portainer CE 2.27.7 LTS The traffic route is Cloudflare -> Traefik -> Docker(portainer-ce).

Fixed by adding these 2 labels into docker-compose.yml for removing Origin header by Traefik.

    labels:
       # Frontend
       # Omit other labels, you need to modify for using your environment.
       - "traefik.http.middlewares.portainer-header.headers.customrequestheaders.Origin="
       - "traefik.http.routers.frontend.middlewares=portainer-header"

This code is for the docker-compose.yml sample below. https://docs.portainer.io/advanced/reverse-proxy/traefik

After setting the following headers in our nginx configuration, Portainer started to work. Thank you.

proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Real-IP $remote_addr;

@Nick-Portainer @oscarzhou-portainer Just tried 2.31.3 STS with my unusual http->https reverse proxy set up and after setting the new TRUSTED_ORIGINS env variable, it is working again. Thanks for the user engagement on this issue and listening to our feedback.

Hi everyone, The team has added a new command-line option: --trusted-origins. This flag offers a unified solution to all the issues outlined above and is planned for inclusion in the upcoming STS release, version 2.32.0. Documentation for this flag will be updated as part of the release. Please stay tuned.

Special thanks to @pmharris77 and @Mister-Hope for providing detailed information that helped the team reproduce and resolve the issues.

Amazing, thanks @oscarzhou-portainer, look forward to testing it out!

Hi everyone, The team has added a new command-line option: --trusted-origins. This flag offers a unified solution to all the issues outlined above and is planned for inclusion in the upcoming STS release, version 2.32.0. Documentation for this flag will be updated as part of the release. Please stay tuned.

Special thanks to @pmharris77 and @Mister-Hope for providing detailed information that helped the team reproduce and resolve the issues.

None of the above fixes worked for me, only way to do anything with Portainer now is by using the direct ports, kind of defeats the purpose of a reverse proxy.

Same problem here. Fixed it by modifying nginx proxy:

location / {
        proxy_set_header Origin "";
        proxy_set_header Referer "portainer:9000";
        proxy_pass http://portainer:9000;
    }

Confirmed this fixed my issue on Portainer CE 2.27.7 LTS

Same here.

Same problem here on: Server Version: 2.27.7 LTS

Same here. Login works after cleaning cookies and cache, but only Login is possible, the rest is not working. proxy_set_header Origin “”; is not working as well. On Url:port works everything, but without certificate and reverse proxy.

I’ve faced similar issue with portainer CE 2.27.7 (I wasn’t able to add new environment) and after reading the topic I’ve fixed my haproxy setup - maybe it will help someone.

I just added following lines to my backend config (remove -->):

backend portainer
  mode http
  --> option forwardfor
  --> http-request add-header X-Forwarded-Proto https if { ssl_fc }
  server portainer 192.168.10.10:9000 check inter 5s

Hey everyone experiencing issues with Home Assistant OS, the Portainer Home Assistant add-on repository https://github.com/alexbelgium/hassio-addons was updated a few hours ago, and the team has verified that the issue is resolved. While the fix isn’t ideal, it’s sufficient for now. Please check it out and take action as needed.

Just for a feedback : Having the same issue with an Apache reverse proxy container in the same configuration, hardcoding the “Origin” header fixed the issue (it’s ugly but it work) :

<VirtualHost *:443>
  ServerName portainer.local:443
  SSLEngine on
  SSLCertificateFile "/usr/local/apache2/conf/ssl/portainer.local.crt"
  SSLCertificateKeyFile "/usr/local/apache2/conf/ssl/portainer.local.key"
  ProxyRequests off
  <Location "/">
    RequestHeader set "Origin" "http://portainer:9000"
    ProxyAddHeaders On
    ProxyPass "http://portainer:9000/"
    ProxyPassReverse "http://portainer:9000/"
  </Location>
</VirtualHost>

can you try adding use-forwarded-headers: "true" as below

controller:
  config:
    use-forwarded-headers: "true"
    ssl-redirect: "false"

The key issue is CSRF protection in Portainer, combined with:

  • AWS NLB terminates SSL at the edge.

  • Requests from the browser have an Origin: https://mycompanycom.

  • But Portainer (via nginx ingress) sees the request as coming over HTTP (X-Forwarded-Proto: http).

  • This mismatch (https in Origin vs http internally) causes Portainer to reject the request.

Chiming in to say I’m having a similar if not the same issue right now.

I’m using the Portainer Home Assistant AddOn which essentially reverse-proxies the site to be shown inside the Home Assistant tab, so it seems to be related. At first I got a few errors where I couldn’t create a Stack, afterwards tried restarting containers which resulted in “Forbidden - Origin Invalid”, then restarted the AddOn and now I’m stuck on the “Unable to login” notification.

UPDATE: Going directly to the exposed port of said AddOn and loggin in there works just fine, so it really just seems to be reverse proxy related. UPDATE 2: Spoke too soon… only the initial login on the exposed port seemed to work, it’s still giving me the “Origin Invalid” errors.

Okay, so I figured out what was wrong with my Traefik setup.

Somehow I had set passHostHeader: false in my Traefik config. Not sure why though. 🤔 🤷🏻 Ref: https://doc.traefik.io/traefik/routing/services/#pass-host-header

After removing that, it has started working for me.

Thanks for the hints, folks 🙇🏻

Thanks for the hint from you too, in IIS, after enabling preserveHostHeader, it’s also working again 🚀

Okay, so I figured out what was wrong with my Traefik setup.

Somehow I had set passHostHeader: false in my Traefik config. Not sure why though. 🤔 🤷🏻 Ref: https://doc.traefik.io/traefik/routing/services/#pass-host-header

After removing that, it has started working for me.

Thanks for the hints, folks 🙇🏻

I’m using IIS as reverse proxy, it works with everything OK, except now with Portainer, which also worked before.

Hey @Nuklon, Could you share more details about your reverse proxy setup, similar to the explanation provided by @pmharris77 ?

Sure, I have portainer running inside a VM (which has a fixed IP).

This:

services:
  portainer:
    image: portainer/portainer-ee:alpine-sts
    restart: unless-stopped
    environment:
      - VIRTUAL_HOST=p.thedomain.com
      - VIRTUAL_PORT=6123
    ports:
      - 9800:9000

In IIS I have a rewrite rule which just rewrites any URL coming from the bound domain to http://vm-ip:9800.

From further investigation, after clearing all cookies (this seems to make a difference), direct IP address connections now appear to work, it is just the HTTP=>HTTPS reverse proxy setup that now does not work after the update.

Presumably the 2.30.0 update (or internal Go update) goes further and requires client to proxy protocol to match portainer server protocol (i.e. http -> http or https -> https), or worse, enforces https between client and proxy?

@Nick-Portainer Please can you confirm what has been implemented in 2.30.0?

What would be nicer is if, in addition to X-Forwarded-Host header checks against VIRTUAL_HOST env variable, there could be a VIRTUAL_HOST_PROTOCOL env variable that could be set and then that could checked against X-Forwarded-Proto header to determine origin validity.

At the very least, if this is not possible, the docs at https://docs.portainer.io/advanced/reverse-proxy should be updated to explain any new limitations on proxy protocols.

Full results below:

Direct connections:

https via IP address:9443 = OK http via IP address:9000 = OK http via local.domain:9000 = OK https via local.domain:9443 = OK

Reverse proxied connections:

HTTP=>HTTPS via local.domain= 403 invalid origin -> WAS WORKING IN 2.29.2 HTTP => HTTP via local.domain= OK HTTPS=>HTTPS via local.domain= Untested