google-auth-library-nodejs: IAP rejects requests from other GAE

I’ve been trying for way too long to get this to work, so I apologize if any rudeness or mistakes come from this:

I’m trying to send a PUT request from one App Engine instance to another App Engine instance. I am getting a client like so:

const client = await auth.getClient();

Correct me if I’m wrong, but that should give me a client with the default GAE service account when run in GAE.

I then send a request like so:

await client.request({
  method: 'PUT',
  url:
    'https://myservice-dot-my-project.appspot.com/api/cool-endpoint?myvar=value',
  data: { some, body, once, told, me },
});

I also tried setting target_audience before the request (although it gives a type error):

client.additionalClaims = {
  target_audience:
    '133742036069-s0m3numb3rs4nd13773rz.apps.googleusercontent.com',
};

With or without the audience claim, I get this thrown in my head: GaxiosError: Invalid IAP credentials: Base64 decode failed on token: ya29.c.<redacted - looks like base64, but weird letters come out when trying to decode>

Probably the most annoying thing is that I can get it to work with auth.fromJSON and providing the json with keys locally. But this is not an option, as the team doesn’t want to manage json files for all our apps in all environments.

I have gone through tons of documentation, SO questions, GitHub issues… There’s no signs that anyone has seen this issue before - or even tried to use the default service account given by app engine when going through IAP.

Please help, I’m getting desperate.


Environment details

  • App Engine Flex
  • Node.js version: latest
  • npm version: latest
  • google-auth-library version: latest

Steps to reproduce

  1. Make an endpoint of some sort with IAP protection
  2. Copy paste code snippets above and insert said endpoint in url
  3. Deploy on GAE flex

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 34 (19 by maintainers)

Most upvoted comments

One workaround if you absolutely need to use a GCE instance is to invoke the IAM API generateIdToken

its an awkward flow where a service account uses an API “on itself” to sign something or to get an id_token.

I woudn’t recommend this as a long term solution (its far better if gce can return the email and/or IAP accepts and decodes the sub claim)

Anyway, to use this flow, you’ll need to manaully do the exchange since this isn’t included in the library to directly do the exchange (though the IAM api is the basis for this pr for impersonated credentials )

export TOKEN=`curl -s -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" | jq -r '.access_token'`export SA_EMAIL=`curl -s -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email"`
export SA_EMAIL=`curl -s -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email"`

curl -s -H "Authorization: Bearer $TOKEN" \ 
   --header 'Content-Type: application/json' \
   -d '{  "audience": "https://foo.bar", "includeEmail": "true" }'\  
  https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/$SA_EMAIL:generateIdToken

i think iap looks for the email field in in the token. if you use a json service accont, the token is populated

if you run the following snippet in a gce instance, it’ll fail against IAP

async function mainIAP() {
  const u = 'https://bmineral-minutia-820.appspot.com';
  const a = '1071284184436-en7pnh3e250v6p2r0ekredacted.apps.googleusercontent.com';
  const auth = new GoogleAuth();
  const client = await auth.getIdTokenClient(
    a
  );
  const res = await client.request({
    method: 'GET',
    url: u,
  });
  console.log(res.data);
}

will error with

 data: "Invalid IAP credentials: JWT 'email' claim isn't a string",

however, if you edit the following bit inline and add on &format=full (which you can read about here),

https://github.com/googleapis/google-auth-library-nodejs/blob/master/src/auth/computeclient.ts#L112

then an id token from gce will work against iap (a service account json file will work since the exchanged id_token has it already).


now, i don’t know if the token returned by cloud run or gae v2 (or gcf) has the email part in it (you can verify by printing the id token and decoding it at jwt.io)

I guess so. It shouldn’t be an issue for us since it’s all internal communication 😃

While format=full will incude the email claim, it’ll also throw in a whole bunch of other stuff too like configuration of the GCE instance (which is gonna expose more data in the easily decodeable jwt token than is necessary. I’m going to file an internal bug to see if the IAP check can use the sub filed in the claim:

if i go on a GCE instance and run

$ curl -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=1071284184436-en7pnh3e250v6p2r0ekue05ekg30vfbh.apps.googleusercontent.com"

i’ll get a jwt that i can decode at jwt.io:

{
  "aud": "1071284184436-en7pnh3e250v6p2r0ekue05ekg30vfbh.apps.googleusercontent.com",
  "azp": "100147106996764479085",
  "exp": 1579189692,
  "iat": 1579186092,
  "iss": "https://accounts.google.com",
  "sub": "100147106996764479085"
}

the sub value is actually the encoded email for the GCE instance’s service account. THat value is easily decoded by any number of other systems at the perimeter like Cloud Run, Cloud Functions, etc…its jus that IAP seems to look for plain email as the claim.

I’ll cc you on the internal bug but my 2c is holding off on the change above to make &format=full default for now (because if IAP can look for sub, then the code youv’e got here would work)…also, i can’t confirm what claims the token has in cloud run, gcf, etc (i’ll ask a collegue now about that)

With the code in master, you can now do this:

const client = await auth.getIdTokenClient(
  '133742036069-s0m3numb3rs4nd13773rz.apps.googleusercontent.com'
);
await client.request({
  method: 'PUT',
  url: 'https://myservice-dot-my-project.appspot.com/api/cool-endpoint?myvar=value',
  data: { some, body, once, told, me },
});

And as long as you’ve set the GOOGLE_APPLICATION_CREDENTIALS environment variable (or are making the call from GCE/AppEngine/GKE/Cloud Run), the client will populate the Authorization header with an ID token.

Ahh I found a similar issue here, https://github.com/cloudendpoints/esp/issues/675, and indeed the way it was fixed is by including format=full in the metadata request. This seems to only effect ID tokens retrieved from GCE.

I’ll add format=full to the metadata querystring, and this should fix the issue.

👋 @AndyClausen this is now released to npm; closing this issue, but please feel free to reopen if you bump into any problems with the implementation – excited to have you try it out.

@AndyClausen I think we have enough info 👍 I have some good news, which is that we have some folks on the team starting to specifically work on auth related issues … I will point them in the direction of this.