gcsfs: gcsfs can't access google storage file, switches to anon user

I am trying to read a csv file present in gs bucket directly into dataframe. I used gcloud auth login to authorize account. But when I try to use gcsfs it thorws: gcsfs.utils.HttpError: Anonymous caller does not have storage.objects.list access to <my-bucket>

Now I ca list/download my files using gsutil. I can also list these files by passing adc file as token to fs=gcsfs.GCSFileSystem(token='adc.json'). My issue is how do I set it as default. Is there a particular location where I put this file. I couldn’t find anything in docs.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 16 (7 by maintainers)

Most upvoted comments

#255 should fix this issue.

Following code now works just fine:

from gcsfs.core import GCSFileSystem
gcs = GCSFileSystem('<project_id>', token='/Users/<username>/.config/gcloud/application_default_credentials.json')
gcs.ls('<bucket>')

FYI, to use default GCP authentication fallback mechanism you can use following:

from gcsfs.core import GCSFileSystem
import google.auth

credentials, _ =google.auth.default()
gcs = GCSFileSystem('<project_id>', token=credentials)
gcs.ls('<bucket>')

vlasenkoalexey on May 6, 2020

See here for how google.auth tries to read your credentials. Enabling application-default worked for me: gcloud auth application-default login

Gabsha on Apr 1, 2020