netbird: Error: token invalid 401

Describe the problem

After upgrading from v0.14.4 to v0.21.3 I am unable to access the dashboard and get this error:

I suspect this is related to the new IdP changes but I am not sure what.

To Reproduce Steps to reproduce the behavior:

1. Pull latest `https://github.com/netbirdio/netbird.git`

2. Copy `infrastructure_files/setup.env.template` to `infrastructure_files/setup.env` (overwrite existing file from old version)

3. Fill in correct values noting these changes:
   
   1. Generate a client secret for the `netbird-client` in Keycloak and `NETBIRD_AUTH_CLIENT_SECRET`
   2. Create a new client `netbird-backend` and follow the steps here: https://docs.netbird.io/selfhosted/identity-providers#step-8-create-a-net-bird-backend-client

4. Re-run `./configure.sh`

5. Run `docker compose pull && docker compose down && docker compose up -d`

Expected behavior

Things work as they used to

Additional context

I tried setting NETBIRD_MGMT_IDP="none" and the dashboard loads but no clients can connect with this error:

netbird up --management-url https://netbird.XXX:33073 --admin-url https://netbird.XXX:443
Error: login failed: rpc error: code = NotFound desc = no SSO provider returned from management. If you are using hosting Netbird see documentation at https://github.com/netbirdio/netbird/tree/main/management for details

i have the same error with a different return code:

Request failed with status code 401. Please refresh the page if the issue continues.
token invalid

docker management logs:

2023-06-28T23:33:07Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
2023-06-28T23:33:07Z INFO management/server/account.go:638: single account mode enabled, accounts number 0
2023-06-28T23:33:07Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
2023-06-28T23:33:07Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443
2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401

that’s my setup.env:

## example file, you can copy this file to setup.env and update its values
##
# Dashboard domain. e.g. app.mydomain.com
NETBIRD_DOMAIN="XXXXX"

# -------------------------------------------
# OIDC
#  e.g., https://example.eu.auth0.com/.well-known/openid-configuration
# ------------------------------------------
NETBIRD_AUTH_DEVICE_AUTH_PROVIDER=hosted
NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT="https://XXXXX/auth/realms/netbird/.well-known/openid-configuration"
NETBIRD_USE_AUTH0=false
NETBIRD_AUTH_CLIENT_ID="netbird-client"
NETBIRD_AUTH_AUDIENCE="netbird-client"
NETBIRD_AUTH_DEVICE_AUTH_CLIENT_ID="netbird-client"
NETBIRD_MGMT_IDP="keycloak"
NETBIRD_IDP_MGMT_CLIENT_ID="netbird-backend"
NETBIRD_IDP_MGMT_CLIENT_SECRET="XXXXX"
NETBIRD_IDP_MGMT_EXTRA_ADMIN_ENDPOINT="https://XXXXX/auth/realms/netbird"

# -------------------------------------------
# Letsencrypt
# -------------------------------------------
# Disable letsencrypt
#  if disabled, cannot use HTTPS anymore and requires setting up a reverse-proxy to do it instead
# e.g. hello@mydomain.com
NETBIRD_LETSENCRYPT_EMAIL="XXXXX"
# -------------------------------------------
# Extra settings
# -------------------------------------------
# Disable anonymous metrics collection, see more information at https://netbird.io/docs/FAQ/metrics-collection
NETBIRD_DISABLE_ANONYMOUS_METRICS=false
# DNS DOMAIN configures the domain name used for peer resolution. By default it is netbird.selfhosted
NETBIRD_MGMT_DNS_DOMAIN=netbird.selfhosted

when i try to connect in the client side:

sudo netbird up --management-url https://XXXXXXXX:33073
Error: login failed: rpc error: code = NotFound desc = no SSO provider returned from management. If you are using hosting Netbird see documentation at https://github.com/netbirdio/netbird/tree/main/management for details

for the netbird-backend, i didn’t do the management steps because, the services accounts roles tab is missing

new management logs:

2023-06-28T23:33:07Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
2023-06-28T23:33:07Z INFO management/server/account.go:638: single account mode enabled, accounts number 0
2023-06-28T23:33:07Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
2023-06-28T23:33:07Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443
2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3101701186: GET /api/users status 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 335186381: GET /api/peers status 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 449313820: GET /api/groups status 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 780097130: GET /api/users status 401
2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3241716118: GET /api/peers status 401
2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 376855912: GET /api/users status 401
2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3584425485: GET /api/groups status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 997841889: GET /api/users status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3647231711: GET /api/users status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2307183365: GET /api/peers status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 418149988: GET /api/groups status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3370837239: GET /api/groups status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2633305945: GET /api/users status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1883989961: GET /api/users?service_user=false status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1727792694: GET /api/groups status 401
2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 127726435: GET /api/users status 401
2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1972559671: GET /api/groups status 401
2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3935102461: GET /api/peers status 401
2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 549383702: GET /api/peers status 401
2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3104191757: GET /api/users status 401
2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3006261035: GET /api/groups status 401
2023-06-29T00:30:18Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w=
2023-06-29T00:30:21Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w=
2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2288542812: GET /api/users status 401
2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1587588152: GET /api/peers status 401
2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 584506530: GET /api/groups status 401
2023-06-29T00:33:55Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w=

_Originally posted by @UncleJ4ck in https://github.com/netbirdio/netbird/issues/959#issuecomment-1612271029_