firebase-admin-node: Firebase appcheck token verification fails because the sdk can't retrieve JWKS

Step 2: Describe your environment

  • Operating System version: macos ventura 13.4
  • Firebase SDK version: 11.10.1
  • Firebase Product: appcheck
  • Node.js version: 16
  • NPM version: 8.18

Step 3: Describe the problem

When the sdk trys to fetch the jwks from google https://firebaseappcheck.googleapis.com/v1/jwks to verify the appcheck token it errors because the endpoint is returning 403 forbidden. It seems that google has added this rate limit for non authenticated identities lately. It can only be bypassed by passing a X-goog-api-key. In the sdk code for appcheck there is no api key passed nor is there the ability to pass an api key as an enduser.

ERROR:

Error: Error fetching Json Web Keys: Forbidden
    at AppCheckTokenVerifier.mapJwtErrorToAppCheckError (/opt/nodejs/node_modules/firebase-admin/lib/app-check/token-verifier.js:144:16)
    at /opt/nodejs/node_modules/firebase-admin/lib/app-check/token-verifier.js:119:24
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Runtime.handler (webpack:///./src/functions/authorization/appcheck/index.ts?:39:13)

I double checked and ran a couple of requests in postman against the jwks endpoint and received a forbidden error as well after some time.

Steps to reproduce:

  1. setup firebase admin sdk
  2. try to verify multiple appcheck tokens

Relevant Code:


const appCheckVerification = async (req, res, next) => {
    const appCheckToken = req.header("X-Firebase-AppCheck");

    if (!appCheckToken) {
        res.status(401);
        return next("Unauthorized");
    }

    try {
        const appCheckClaims = await getAppCheck().verifyToken(appCheckToken); !!!FAILS HERE!!!

        // If verifyToken() succeeds, continue with the next middleware
        // function in the stack.
        return next();
    } catch (err) {
        res.status(401);
        return next("Unauthorized");
    }
}

Using the sdk to verify an appcheck token does not work after hitting the rate limit of the jwks endpoint. It seems like an api key header needs to be passed in this line: https://github.com/firebase/firebase-admin-node/blob/a4019e491c9d36167853eaba056adebd0e9dfc87/src/utils/jwt.ts#L61

About this issue

  • Original URL
  • State: closed
  • Created 9 months ago
  • Reactions: 21
  • Comments: 27

Most upvoted comments

Thank you everyone for swiftly identifying and reporting this issue. We have identified the issue and we are rolling out a fix now; once the fix is rolled out this issue should be resolved. This error affected a small portion of our users as we have gradual rollouts in place.

We really understand the frustration and we apologize for the inconvenience caused. We have added additional testing for the JWKS endpoint to prevent future recurrence.

Thank you for your patience everyone. We have completed the rollout of the fix. Please let us know if you continue to experience issues.

We apologize again for the frustration this has caused.

This was a disastrous change for code I had running in production. This change breaks the official SDKs’ initialization, and prevented appcheck from working at all. I had to patch the SDK to include an API key in the URL in order to get my server running again, e.g. https://firebaseappcheck.googleapis.com/v1/jwks?key=<firebase web API key> but this is really an unacceptable breaking change.

Please either document this change and update the SDKs, or revert it

It happened to us on europe-central2 deployment. It keeps working on europe-west, please don’t release this change further 🙏

same here 😢

I’m using the firebase-admin-python, but I have the same issue. On my local machine I’m able to retrieve the jwks, but on a AWS server, it returns a 403.