aiogoogle: Token is not getting refreshed after its expired after around 1 hour
Issue: I’ve been executing a script to fetch all the objects from cloud storage. There are around a lot of objects present in the bucket. Observation: The script fetches half of the objects in around 1 hr and then raises the below-mentioned error. Traceback: Traceback (most recent call last):
File "~/aiogoogle/models.py", line 341, in _next_page_generator
prev_res = await sess.send(next_req, full_res=True)
File "~/aiogoogle/sessions/aiohttp_session.py", line 183, in send
results = await schedule_tasks()
File "~/aiogoogle/sessions/aiohttp_session.py", line 175, in schedule_tasks
return await asyncio.gather(*tasks, return_exceptions=False)
File "~/aiogoogle/sessions/aiohttp_session.py", line 157, in get_response
response.raise_for_status()
File "~/aiogoogle/models.py", line 453, in raise_for_status
raise AuthError(msg=self.reason, req=self.req, res=self)
aiogoogle.excs.AuthError:
Unauthorized
Content:
{'code': 401,
'errors': [{'domain': 'global',
'location': 'Authorization',
'locationType': 'header',
'message': 'Invalid Credentials',
'reason': 'authError'}],
'message': 'Invalid Credentials'}
Request URL:
https://storage.googleapis.com/storage/v1/b/....
Question: Is there any other way to reset the access token present in the session_factory’s send() method? or am I missing a step?
code snippet:
blobs_full_request = await google_client.as_service_account(
storage_client.objects.list(bucket=BUCKET),
full_res=True,
)
async for blob_page in blobs_full_request:
yield blob_page
Possible issue: When pagination starts, if there are more pages to go through, the package fails to look for the validity of the access_token.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (17 by maintainers)
Hey, thanks for the detailed ticket!
Yes, this indeed seems to be the case.
I’ll be grateful if you can create a PR with a fix 🙏 and I’ll go ahead and create a new release once you open the PR and it’s merged.
Thanks!
Thanks for the clarification @neel004!
I think we should avoid setting user_creds to auth manager if the user didn’t explicitly do so when instantiating the auth manager, otherwise, it will be useless to allow one off user_creds requests.
If it will be easy, maybe we can somehow pass the user_creds object like we pass the auth manager?
Thanks for the quick merge, I’ll raise a PR once I have proper fix ready, till then we can keep the issue open.
Thank you so much @neel004! I’ve just released v4.4.0
p.s. I think right now we’re refreshing tokens on every next page request, ideally, we should check if the access token is expired first to avoid the extra network call. We probably need to work on the Auth.ServiceAccountManager.refresh method. But I guess this isn’t urgent, hence why I merged and released it.
Hey, @omarryhan @praveen-elastic I’ve raised a PR which fixes this issue for service-accounts. @omarryhan can you please go ahead and release this, meanwhile I’ll raise subsequent PRs for other auth modes.