aiogoogle: Token is not getting refreshed after its expired after around 1 hour

Issue: I’ve been executing a script to fetch all the objects from cloud storage. There are around a lot of objects present in the bucket. Observation: The script fetches half of the objects in around 1 hr and then raises the below-mentioned error. Traceback: Traceback (most recent call last):

  File "~/aiogoogle/models.py", line 341, in _next_page_generator
    prev_res = await sess.send(next_req, full_res=True)
  File "~/aiogoogle/sessions/aiohttp_session.py", line 183, in send
    results = await schedule_tasks()
  File "~/aiogoogle/sessions/aiohttp_session.py", line 175, in schedule_tasks
    return await asyncio.gather(*tasks, return_exceptions=False)
  File "~/aiogoogle/sessions/aiohttp_session.py", line 157, in get_response
    response.raise_for_status()
  File "~/aiogoogle/models.py", line 453, in raise_for_status
    raise AuthError(msg=self.reason, req=self.req, res=self)
aiogoogle.excs.AuthError: 



Unauthorized

Content:
{'code': 401,
 'errors': [{'domain': 'global',
             'location': 'Authorization',
             'locationType': 'header',
             'message': 'Invalid Credentials',
             'reason': 'authError'}],
 'message': 'Invalid Credentials'}

Request URL:
https://storage.googleapis.com/storage/v1/b/....

Question: Is there any other way to reset the access token present in the session_factory’s send() method? or am I missing a step?

code snippet:

blobs_full_request = await google_client.as_service_account(
    storage_client.objects.list(bucket=BUCKET),
    full_res=True,
)
async for blob_page in blobs_full_request:
    yield blob_page

Possible issue: When pagination starts, if there are more pages to go through, the package fails to look for the validity of the access_token.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (17 by maintainers)

Most upvoted comments

Hey, thanks for the detailed ticket!

Possible issue: When pagination starts, if there are more pages to go through, the package fails to look for the validity of the access_token.

Yes, this indeed seems to be the case.

I’ll be grateful if you can create a PR with a fix 🙏 and I’ll go ahead and create a new release once you open the PR and it’s merged.

Thanks!

Thanks for the clarification @neel004!

I think we should avoid setting user_creds to auth manager if the user didn’t explicitly do so when instantiating the auth manager, otherwise, it will be useless to allow one off user_creds requests.

If it will be easy, maybe we can somehow pass the user_creds object like we pass the auth manager?

Thanks for the quick merge, I’ll raise a PR once I have proper fix ready, till then we can keep the issue open.

Thank you so much @neel004! I’ve just released v4.4.0

p.s. I think right now we’re refreshing tokens on every next page request, ideally, we should check if the access token is expired first to avoid the extra network call. We probably need to work on the Auth.ServiceAccountManager.refresh method. But I guess this isn’t urgent, hence why I merged and released it.

Hey, @omarryhan @praveen-elastic I’ve raised a PR which fixes this issue for service-accounts. @omarryhan can you please go ahead and release this, meanwhile I’ll raise subsequent PRs for other auth modes.