harbor: Core logs error "failed to verify token: oidc: malformed jwt: square/go-jose: compact JWS format must have three parts" after switch to S3 storage -- excessive warning messages.

Expected behavior and actual behavior:

On a Harbor installation (Kubernetes using harbor-helm), I’m seeing the following error for all requests to the core server:

2020-06-16T11:32:11Z [WARNING] [/server/middleware/security/idtoken.go:45][requestID="a4f074b4-2cd2-4976-a68c-5f378fbc4882"]: failed to verify token: oidc: malformed jwt: square/go-jose: compact JWS format must have three parts
2020/06/16 11:32:11.281 [D] [transaction.go:62]  |    10.217.5.96| 200 |    502.279µs|   match| GET      /api/v2.0/ping   r:/api/v2.0/ping
2020-06-16T11:32:14Z [WARNING] [/server/middleware/security/idtoken.go:45][requestID="29e92515-252d-4218-b16c-7530880cb5a6"]: failed to verify token: oidc: malformed jwt: square/go-jose: compact JWS format must have three parts
2020/06/16 11:32:14.062 [D] [transaction.go:62]  |    10.217.5.96| 200 |    294.838µs|   match| GET      /api/v2.0/ping   r:/api/v2.0/ping

If I switch the installation back to using PersistentVolumeClaims, the errors go away (and leave just the regular ping’s).

Steps to reproduce the problem:

Seem to be related to using a S3 backend, likely in combination with us using oAuth for Single Sign-On.

Versions: Please specify the versions of following systems.

harbor version: 2.0.0
harbor-helm 1.4.0
kubernetes: 1.17.3
oAuth server is dex (https://github.com/dexidp/dex)

Additional context:

(I was unable to locate any meaningful configuration file on the core server pod. Relevant logs reproduced above.)

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 10
Comments: 18 (7 by maintainers)

Commits related to this issue

Reduce warning logs in OIDC middleware If the request does not have bearer token in the header, do not decode the empty string. Fixes #12261 Signed-off-by: Daniel Jiang <jiangd@vmware.com> — committed to reasonerjt/harbor by reasonerjt 3 years ago

Most upvoted comments

+1 with:

Harbor version 2.0.2
OIDC Auth with Keycloak
Ceph S3 Backend

l4z41 on Aug 17, 2020

I’m trying to debug an issue where Harbor 2.1.0 seem to crash whenever I try pull-through caching from Docker hub. Turning the log-level to debug means that every request to /ping logs a stanza like this every five seconds [1]:

2020-09-30T07:28:46Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0051d8e0-2bbf-4c3d-8e93-d13065c5a181 to the logger for the request GET /api/v2.0/ping
2020-09-30T07:28:46Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:52]: In artifact info middleware, url: /api/v2.0/ping
2020-09-30T07:28:46Z [DEBUG] [/common/utils/oidc/helper.go:194]: Raw ID token for verification: 
2020-09-30T07:28:46Z [WARNING] [/server/middleware/security/idtoken.go:45][requestID="0051d8e0-2bbf-4c3d-8e93-d13065c5a181"]: failed to verify token: oidc: malformed jwt: square/go-jose: compact JWS format must have three parts
2020-09-30T07:28:46Z [DEBUG] [/server/middleware/security/unauthorized.go:29][requestID="0051d8e0-2bbf-4c3d-8e93-d13065c5a181"]: an unauthorized security context generated for request GET /api/v2.0/ping

This amount of noise makes it very hard to figure out what’s actually going on (besides unauthenticated ping-requests, that is).

[1] Harbor’s Helm-chart runs liveness- and readiness-probes every 10s each. Both call /api/v2.0/ping.

msiebuhr on Sep 30, 2020