cloudstack: SSO fails with error "Expired session, missing signature, or missing apiKey"

ISSUE TYPE

• Bug Report

COMPONENT NAME

Authentication, SAML2

CLOUDSTACK VERSION

4.16.1

CONFIGURATION

SAML2 authentication is enabled and configured to Microsoft Azure

OS / ENVIRONMENT

Rocky Linux 8.5

SUMMARY

CloudStack SSO works the first time on a clean browser session. However, after the user attempts to log out or when the CloudStack session expires, CloudStack will no longer allow the user to authenticate via SSO any more.

What happens on subsequent SSO sign-in attempts is the user gets sent to the SAML2 server which redirects them back with the token. Everything is good up to here. However, CloudStack for some reason invalidates the session and the user is redirected back to the login page but this time with the ‘Single Sign-On’ option disabled.

The management-server.log shows the SSO working initially.

2022-03-16 12:00:24,504 DEBUG [c.c.a.ApiServlet] (qtp365590665-1149:ctx-a6c7c3f1) (logid:e18c4373) ===START===  ###.###.96.250 -- GET  command=samlSso&idpid=https://sts.windows.net/c609a0ec-a5e3-4631-9686-#########/
2022-03-16 12:00:24,513 DEBUG [o.a.c.a.c.SAML2LoginAPIAuthenticatorCmd] (qtp365590665-1149:ctx-a6c7c3f1) (logid:e18c4373) Sending SAMLRequest id=idj8da8n95krcr1bco9r7h13a2008cfrv4
2022-03-16 12:00:24,560 DEBUG [c.c.a.ApiServlet] (qtp365590665-1149:ctx-a6c7c3f1) (logid:e18c4373) ===END===  ###.###.96.250 -- GET  command=samlSso&idpid=https://sts.windows.net/c609a0ec-a5e3-4631-9686-#########/
2022-03-16 12:00:25,340 DEBUG [c.c.a.ApiServlet] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) ===START===  ###.###.96.250 -- POST  command=samlSso
2022-03-16 12:00:25,369 DEBUG [o.a.c.a.c.SAML2LoginAPIAuthenticatorCmd] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) Received SAMLResponse in response to id=idj8da8n95krcr1bco9r7h13a2008cfrv4
2022-03-16 12:00:25,370 DEBUG [o.a.c.s.SAMLUtils] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) SAML attribute name: http://schemas.microsoft.com/identity/claims/tenantid friendly-name:null value:c609a0ec-a5e3-4631-9686-#########
2022-03-16 12:00:25,370 DEBUG [o.a.c.s.SAMLUtils] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) SAML attribute name: http://schemas.microsoft.com/identity/claims/objectidentifier friendly-name:null value:084f18c2-6e97-4d2e-9e09-4938baf5b6b8
2022-03-16 12:00:25,370 DEBUG [o.a.c.s.SAMLUtils] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) SAML attribute name: http://schemas.microsoft.com/identity/claims/displayname friendly-name:null value:######## ########
2022-03-16 12:00:25,370 DEBUG [o.a.c.s.SAMLUtils] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) SAML attribute name: http://schemas.microsoft.com/identity/claims/identityprovider friendly-name:null value:https://sts.windows.net/c609a0ec-a5e3-4631-9686-#########/
2022-03-16 12:00:25,370 DEBUG [o.a.c.s.SAMLUtils] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) SAML attribute name: http://schemas.microsoft.com/claims/authnmethodsreferences friendly-name:null value:urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
2022-03-16 12:00:25,370 DEBUG [o.a.c.s.SAMLUtils] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) SAML attribute name: http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname friendly-name:null value:########
2022-03-16 12:00:25,370 DEBUG [o.a.c.s.SAMLUtils] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) SAML attribute name: http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname friendly-name:null value:########
2022-03-16 12:00:25,370 DEBUG [o.a.c.s.SAMLUtils] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) SAML attribute name: http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress friendly-name:null value:#########@##################
2022-03-16 12:00:25,370 DEBUG [o.a.c.s.SAMLUtils] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) SAML attribute name: http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name friendly-name:null value:#########@##################
2022-03-16 12:00:25,374 DEBUG [c.c.u.AccountManagerImpl] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) Attempting to log in user: #########@################## in domain 1
2022-03-16 12:00:25,385 DEBUG [o.a.c.s.SAML2UserAuthenticator] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) Trying SAML2 auth for user: #########@##################
2022-03-16 12:00:25,391 DEBUG [c.c.u.AccountManagerImpl] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) CIDRs from which account 'Acct[403fe246-993a-4bec-8602-f935de97ea26-RCS] -- Account {"id": 7, "name": "RCS", "uuid": "403fe246-993a-4bec-8602-f935de97ea26"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-03-16 12:00:25,391 DEBUG [c.c.u.AccountManagerImpl] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) User: #########@################## in domain 1 has successfully logged in
2022-03-16 12:00:25,397 INFO  [c.c.a.ApiServer] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) Current user logged in under UTC timezone
2022-03-16 12:00:25,397 INFO  [c.c.a.ApiServer] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) Timezone offset from UTC is: 0.0
2022-03-16 12:00:25,399 DEBUG [c.c.a.ApiServlet] (qtp365590665-1149:ctx-43bf32d6) (logid:d2efb5a3) ===END===  ###.###.96.250 -- POST  command=samlSso
2022-03-16 12:00:30,138 DEBUG [c.c.a.ApiServlet] (qtp365590665-1150:ctx-182d1ee5) (logid:c66cc511) ===START===  ###.###.96.251 -- GET  listall=true&command=listZones&response=json
2022-03-16 12:00:30,139 DEBUG [c.c.a.ApiServlet] (qtp365590665-1150:ctx-182d1ee5) (logid:c66cc511) ===END===  ###.###.96.251 -- GET  listall=true&command=listZones&response=json
2022-03-16 12:00:30,145 DEBUG [c.c.a.ApiServlet] (qtp365590665-1149:ctx-05801923) (logid:c7905631) ===START===  ###.###.96.250 -- GET  command=listApis&response=json
2022-03-16 12:00:30,145 DEBUG [c.c.a.ApiServer] (qtp365590665-1149:ctx-05801923 ctx-a4a75c17) (logid:c7905631) Expired session, missing signature, or missing apiKey -- ignoring request. Signature: null, apiKey: null
2022-03-16 12:00:30,145 DEBUG [c.c.a.ApiServlet] (qtp365590665-1149:ctx-05801923 ctx-a4a75c17) (logid:c7905631) ===END===  ###.###.96.250 -- GET  command=listApis&response=json
2022-03-16 12:00:30,159 DEBUG [c.c.a.ApiServlet] (qtp365590665-1155:ctx-627512a2) (logid:8b94f06a) ===START===  ###.###.96.253 -- GET  command=listCapabilities&response=json
2022-03-16 12:00:30,160 DEBUG [c.c.a.ApiServer] (qtp365590665-1155:ctx-627512a2 ctx-92910815) (logid:8b94f06a) Expired session, missing signature, or missing apiKey -- ignoring request. Signature: null, apiKey: null
2022-03-16 12:00:30,160 DEBUG [c.c.a.ApiServlet] (qtp365590665-1157:ctx-1b86257d) (logid:c0d262f3) ===START===  ###.###.96.250 -- GET  username=#########%40##################&command=listUsers&response=json
2022-03-16 12:00:30,160 DEBUG [c.c.a.ApiServer] (qtp365590665-1157:ctx-1b86257d ctx-376c121e) (logid:c0d262f3) Expired session, missing signature, or missing apiKey -- ignoring request. Signature: null, apiKey: null
2022-03-16 12:00:30,160 DEBUG [c.c.a.ApiServlet] (qtp365590665-1155:ctx-627512a2 ctx-92910815) (logid:8b94f06a) ===END===  ###.###.96.253 -- GET  command=listCapabilities&response=json
2022-03-16 12:00:30,160 DEBUG [c.c.a.ApiServlet] (qtp365590665-1157:ctx-1b86257d ctx-376c121e) (logid:c0d262f3) ===END===  ###.###.96.250 -- GET  username=#########%40##################&command=listUsers&response=json
2022-03-16 12:00:30,164 DEBUG [c.c.a.ApiServlet] (qtp365590665-1153:ctx-56829601) (logid:11ce94ee) ===START===  ###.###.96.253 -- GET  command=listLdapConfigurations&response=json
2022-03-16 12:00:30,164 DEBUG [c.c.a.ApiServer] (qtp365590665-1153:ctx-56829601 ctx-afefa441) (logid:11ce94ee) Expired session, missing signature, or missing apiKey -- ignoring request. Signature: null, apiKey: null
2022-03-16 12:00:30,164 DEBUG [c.c.a.ApiServlet] (qtp365590665-1153:ctx-56829601 ctx-afefa441) (logid:11ce94ee) ===END===  ###.###.96.253 -- GET  command=listLdapConfigurations&response=json
2022-03-16 12:00:30,168 DEBUG [c.c.a.ApiServlet] (qtp365590665-1156:ctx-3d6d3ba3) (logid:c3876972) ===START===  ###.###.96.251 -- GET  command=cloudianIsEnabled&response=json
2022-03-16 12:00:30,168 DEBUG [c.c.a.ApiServer] (qtp365590665-1156:ctx-3d6d3ba3 ctx-71161310) (logid:c3876972) Expired session, missing signature, or missing apiKey -- ignoring request. Signature: null, apiKey: null
2022-03-16 12:00:30,168 DEBUG [c.c.a.ApiServlet] (qtp365590665-1156:ctx-3d6d3ba3 ctx-71161310) (logid:c3876972) ===END===  ###.###.96.251 -- GET  command=cloudianIsEnabled&response=json

The work around is to completely wipe all cookies and start over or use CloudStack in an incognito session.

Local accounts work as expected. However, logging in and out with a local account does not fix the SSO issue.

STEPS TO REPRODUCE

1. Log in with Single Sign-On method
2. Once in the management console, click log out or let the session time out
3. On the login page, observe that the SSO option is disabled. Refresh the page with shift+f5 which brings the option back.
4. Click on Single Sign-On and log in again

EXPECTED RESULTS

Expect that subsequent SSO logins should work.

ACTUAL RESULTS

Successful SSO authentication redirect me back to the login page.