nodejs-storage: "Cannot sign data without `client_email`" for getSignedUrl

I’m trying to get a signed url from a storage bucket, but am getting a signing error. I have permission to read from the bucket, and have been able to stream its contents fine.

Code:

  ...

  const expires = new Date()
  expires.setSeconds(expires.getSeconds() + 30)

  const storage = new Storage({ projectId });

  storage
    .bucket(bucketName)
    .file(`${directory}/${packageName}`)
    .getSignedUrl({
      expires,
      action: 'read',
    })
    .then(url => {
      res.send(url)
      console.log(url)
    })
    .catch(console.log)

Error:

{ SigningError: Cannot sign data without `client_email`.
    at /Users/pikelnys/Desktop/code/gcs-project/api/node_modules/@google-cloud/storage/src/file.js:1784:16
    at Auth._signWithApi (/Users/pikelnys/Desktop/code/gcs-project/api/node_modules/google-auto-auth/index.js:331:7)
    at getCredentials (/Users/pikelnys/Desktop/code/gcs-project/api/node_modules/google-auto-auth/index.js:316:14)
    at googleAuthClient.getCredentials (/Users/pikelnys/Desktop/code/gcs-project/api/node_modules/google-auto-auth/index.js:194:9)
    at /Users/pikelnys/Desktop/code/gcs-project/api/node_modules/google-auth-library/build/src/auth/googleauth.js:602:67
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7) message: 'Cannot sign data without `client_email`.' }

Environment details

  • OS: macOS 10.13.4
  • Node.js version: 8.11.2
  • npm version: 6.4.1
  • @google-cloud/storage version: 1.7.0

Edit: I’m getting this in both a local environment (where I authenticate with gcloud auth application-default login) and while the code is deployed to cloud functions.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 20
  • Comments: 48 (12 by maintainers)

Commits related to this issue

Most upvoted comments

Is this possible to authenticate without a keyfile?

Service account isn’t good for multi-stage environments and security. This should be reopened. You should be using the exposed environment that has the same values.

Hi @pikelnys.

You will have to specify keyFilename while creating the Storage object if you are deploying locally.

keyFilename is not required if you have deployed your code under the same project your Google Cloud Bucket was created. You just have to ensure you have the right access by having a look at your service-account-key.

This is well explained here

The error says cannot sign without client_email. client_email is available inside the service-account-key.json but was not provided while creating the object hence the error. This error would not have arrived had you deployed your code inside the same project as your Cloud Bucket was.

Here is a sample service-account-key.json file. Look for client_email in the file.

{
"type": "service_account",
"project_id": "[PROJECT-ID]",
"private_key_id": "[KEY-ID]"
"private_key": "-----BEGIN PRIVATE KEY-----\n[PRIVATE-KEY]\n-----END PRIVATE KEY-----\n",
"client_email": "[SERVICE-ACCOUNT-EMAIL]",
"client_id": "[CLIENT-ID]",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/[SERVICE-ACCOUNT-EMAIL]"
}

Steps to Follow.

Download the service account key in JSON format and specify its path inside keyFilename. Default service-account-key provided by Google has read-only access. So make sure if you want to write to your bucket you would have to change the permissions inside the service-account.

Here is the Google Codelab link for reference

var storage = new Storage({
  projectId: "your project id",
  keyFilename:"your-service-account-key.json"
});

Hope this resolves your issue.

For anyone else that comes across this issue, here is a bit more info for posterity.

The GCP client libraries docs use the Storage libary as the example when leveraging ADC locally. The same docs also call out the fact that using service accounts locally is bad practice (see the Note in this section).

However, for some reason Google seems to contradict itself with the implementation of the getSignedUrl method and its corresponding endpoint. After a couple hours of debugging and re-reading the disparate and disconnected docs scattered all over the place, I found the source of the issue as documented here. V4 (and the legacy V2) signing requires usage of a service account.

It would be nice if Google would have bothered to call this out in the ADC docs where they use Storage as an example, or better yet, return the relevant instructions in the SigningError message, especially because they seemingly contradict themselves within the different docs on related topics.

Hope this helps someone!

I was getting “Cannot sign data without client_email” error when running firebase emulators:start.

I fixed it by setting up a service account and downloading the service account .json credentials file as service_account.json, then running GOOGLE_APPLICATION_CREDENTIALS="service_account.json" firebase emulators:start

BUMP as there isn’t seem to be a way to deal with it via emulator without exposing dev/prod configs.

Got the issue and solved it thanks to this solution firebase-adminsdk-ov3eh@staging-louve.iam.gserviceaccount.com thanks @christiangenco

Jeez. Enabling OIDC has been a massive challenge with the garbage documentation out there.

Anyhow, after hours of fighting, here’s how I could sign the URL using service account impersonation:

1st. Create the service account:

gcloud iam service-accounts create "local-dev-account" \
    --description="Local Development Account" \
    --project="${YOUR_PROJECT_ID}" \
    --display-name="Local Development Account"

2nd. Create the local_dev_role that will be attached to the service account:

gcloud iam roles create "local_dev_role" \
 --project="${YOUR_PROJECT_ID}" \
 --file="./roles-local.gcp.yml"

roles-local.gcp.yml:

# gcloud iam roles update local_dev_role --project=${YOUR_PROJECT_ID} --file=./roles-local.gcp.yml
# https://stackoverflow.com/a/68901952/1057052
# We are currently using Workload Identity Federation to authenticate. This means that there are no service keys passed
# around.
# Therefore, we need another mechanism to authenticate when developing locally.
# In Production, services are authenticated via IAM directly.
# This files contains only the minimum set of permissions required to access the development environment.
# We will generate a local JSON file that will serve as an impersonation to a service account.
# Required roles:

title: Local Development Roles
description: |
  This policy is for local development as Google discourages the usage of service account keys. This will work for impersonation
stage: GA
# https://cloud.google.com/iam/docs/permissions-reference
includedPermissions:
  # Permissions for GCR
  - storage.objects.create
  - storage.objects.delete # Optional: only include if you need to delete images
  - storage.objects.get
  - storage.objects.list
  - storage.objects.update
  - storage.objects.getIamPolicy
  - storage.objects.setIamPolicy
  # Add these permissions for token management
  - iam.serviceAccounts.getAccessToken
  - iam.serviceAccounts.signBlob # Required for signed URLs
  # Add more permissions as needed

3rd. Attach the service account at the project level

gcloud projects add-iam-policy-binding ${YOUR_PROJECT_ID} \
 --member="serviceAccount:local-dev-account@${YOUR_PROJECT_ID}.iam.gserviceaccount.com" \
 --role="projects/${YOUR_PROJECT_ID}/roles/local_dev_role" \
 --project="${YOUR_PROJECT_ID}"

Note: There’s a similar command called gcloud iam service-accounts add-iam-policy-binding… Don’t! This will not work.

4th. Impersonate the service account:

gcloud auth application-default login --impersonate-service-account=local-dev-account@alertdown-staging.iam.gserviceaccount.com

If that didn’t work, and you’re getting errors about the account not getting impersonated:

Attach the roles/iam.serviceAccountTokenCreator to YOUR email address:

gcloud iam service-accounts add-iam-policy-binding \
    local-dev-account@${YOUR_PROJECT_ID}.iam.gserviceaccount.com \
    --member="user:your-logged-in-user@email.com" \
    --role="roles/iam.serviceAccountTokenCreator" \
    --project="${YOUR_PROJECT_ID}"

gcloud iam service-accounts add-iam-policy-binding \
    local-dev-account@${YOUR_PROJECT_ID}.iam.gserviceaccount.com \
    --member="user:your-logged-in-user@email.com" \
    --role="roles/iam.serviceAccountUser" \
    --project="${YOUR_PROJECT_ID}"

And just for the sake of completeness: Here’s the full Storage class wrapper I’ve created in Node:

import {
  InternalServerException,
  PromiseExceptionResult,
} from '@alertdown/core';
import { Storage as GoogleCloudStorage } from '@google-cloud/storage';
import { Err, Ok } from 'oxide.ts';

const getFileUrl = (bucket: string, filename: string) => {
  return `https://storage.googleapis.com/${bucket}/${filename}`;
};

type ConstructorInput = {
  bucket: string;
  /**
   * Development only
   */
  projectId?: string;
};

export class Storage {
  #client: GoogleCloudStorage;
  #bucket: string;

  constructor(input: ConstructorInput) {
    this.#client = new GoogleCloudStorage({
      projectId: input.projectId,
    });
    this.#bucket = input.bucket;
  }

  async upload(file: Buffer, filename: string): PromiseExceptionResult<string> {
    try {
      const bucket = this.#client.bucket(this.#bucket);
      const blob = bucket.file(filename);

      await blob.save(file, {
        metadata: {
          contentType: 'image/png',
        },
      });
      return Ok(getFileUrl(this.#bucket, filename));
    } catch (error) {
      return Err(new InternalServerException(error as Error));
    }
  }

  async resolveUrl(url: string): PromiseExceptionResult<string> {
    try {
      const parsedUrl = new URL(url);
      const filename = parsedUrl.pathname.split('/').pop();
      console.log((await this.#client.authClient.getClient()).credentials);

      if (!filename) {
        return Err(new InternalServerException('Invalid URL format'));
      }

      const bucket = this.#client.bucket(this.#bucket);
      const file = bucket.file(filename);
      const [signedUrl] = await file.getSignedUrl({
        version: 'v4',
        action: 'read',
        expires: Date.now() + 15 * 60 * 1000, // 15 minutes
      });
      return Ok(signedUrl);
    } catch (error) {
      console.error(`Error resolving URL: ${url}`, error);
      return Err(new InternalServerException(error as Error));
    }
  }
}

Note: See that I only pass the projectId (YOUR_PROJECT_ID) and the bucket name.

Additional Notes:

  • List of Permissions - Google Cloud
  • [View Roles Permissionshttps://console.cloud.google.com/iam-admin/roles) (You can view the individual roles permissions here. Then you can add it to your yam file as needed)
  • Every time you update a permission, if it doesn’t work for any reason, try reauthenticating.

Helpful Commands For Debugging:

  1. Check if the role exists and its permissions
gcloud iam roles describe local_dev_role \
    --project=${YOUR_PROJECT_ID}
  1. Verify if the role is attached to the service account:
gcloud projects get-iam-policy ${YOUR_PROJECT_ID} \
    --filter="bindings.members:local-dev-account@${YOUR_PROJECT_ID}.iam.gserviceaccount.com" \
    --format="table(bindings.role)"
  1. Command to update the role from the yaml (roles-local-gcp.yml)
gcloud iam roles update local_dev_role \
    --project=${YOUR_PROJECT_ID} \
    --file=./roles-local.gcp.yml

Just realized the previously linked issue was merged in last year and the client_email issue was resolved (no longer required) for impersonated service accounts. I just updated my library version and can confirm no workaround is necessary anymore. Adding my version info for posterity:

Google Cloud SDK 495.0.0
bq 2.1.8
core 2024.09.27
gcloud-crc32c 1.0.0
gsutil 5.30
Node.js v20.11.0

"@google-cloud/storage": "^7.13.0"

bump. this should be re-opened.

I can’t believe it’s been six years since I ran into this issue and it’s still not resolved. Since then, I found a work around for the issue, the project died, I quit that job, and the company went out of business.

For me it is while using firebase emulator, I just want to be able to sign local storage URLs. Nothing cloud side. Everything just local!

Bump. Firebase emulator should emulate the environment, this appears to be a failure state

That makes total sense. Have any issues been submitted to anyone else for this? I’d love to take this discussion to someone else’s repo instead who could actually fix it. I think you’re in the best position to write up what needs to be fixed in which repo and also influence the rest of the googleapis team.

In the meantime, you’ve got a known broken use case and the errors that are bubbling up out of your library don’t make sense in the context of your library. Is that a valid bug worth having an open issue for?

@dinkel-katilyst I found this public issue that was filed against gcloud cli. For yourself and anyone else, I would suggest making additional noise on that issue to try and get it visible again. In the meantime I found the same issue internally and I will reach out to the team and link them to this issue. Can’t promise anything, but I’ll try what I can to get this either fixed or rationale why it was never done.

I see a few bumps on this old issue so wanted to leave a comment. It seems that the issue most are bumping about is trying to sign using ADC which does not include the client_email field. Unfortunately, this library does not own the JSON file produced during login nor the signing process. Signing with impersonation should now work and might be one alternative to explore.

If there is a different issue, I would ask that you open a new issue so that I can investigate / resolve as appropriate.

I don’t think you should switch to a json keyfile. I just added an npm command to add the client_email to the application default credentials after login using the json lib.

npm install --save-dev json
"scripts": {
  "login": "gcloud auth application-default login --impersonate-service-account={MY_EMAIL} {EXTRA_ARGUMENTS}  && npm run update-client-email",
  "update-client-email": "json -I -f ~/.config/gcloud/application_default_credentials.json -e 'this.client_email=\"{MY_EMAIL}\"'"
}

I haven’t deployed yet to Google Cloud, but I think you should be able to run this login command locally only, and it should just work when deployed to cloud?

Edit: added --impersonate-service-account as a required argument

I recently wrote a Medium article describing a workaround I created for local development. There is also a git repo with my solution. Article | Repo I use IAM Credentials API coupled with a cloud function. Hope this helps someone.

Circling back. I spoke with the teams who own the login / writing of the ADC JSON file. This isn’t just a matter of adding a client_email to the JSON as this alone would not allow for signing. It would require involvement of HMAC keys and from what I was told this is beyond the scope of ADC.

The best course of action is to utilize SA impersonation for signing in these situations.

Checking the application_default_credentials.json file, it still doesn’t have a client_email value in it.

My hunt for a working solution continues. 🤕

@dinkel-katilyst if you just add client_email of the impersonated service account after signing in (which will clear the client_email), this is the quickest workaround. I automated it as a single npm command in my previous solution

Any workaround?

Double bump. There’s not a great firebase-admin solution for this when running in the emulator without using a service account json. It’s messy, insecure, and inconvenient for testing…

It’s really easy on the frontend with Firebase Auth. Ex:

  const imageListRef = ref(storage, `${currentUser?.uid}/story_images/${story.id}.png`);
  try {
      story.imageBlobUri = await getDownloadURL(imageListRef);
  } catch (error) {
      story.imageBlobUri = undefined;
  }
  readyStories.push(story);

@howlettga I have tried using the solution you suggest by adding in a client_email field to the existing ADC, but that doesn’t seem to work now. Were you able to get it to work?

The only way I can get it working currently is by passing in a service account key when using locally to generate a Signed URL.

Thanks @tritone, appreciate the response!

Hey @timbuckiii and @ollyde , just wanted to note a couple things:

  1. GCS does in fact support signing without having a local service account key file by using a GCE workload identity or service account impersonation to call signBlob. I definitely see that this could be clearer in the GCS docs and will file an internal issue to improve this.
  2. The nodejs-storage library in particular does not yet have support for signing via service account impersonation. However, there is an open issue for this in the node auth library. Comment or follow along at googleapis/google-auth-library-nodejs#1443

Also, in the future, if you could open a new issue for any problems rather than commenting on a closed one, that would help us respond faster. Thanks!