infrastructure: [Non active][Incident] openscapes: some Chrome users unable to login with GitHub authentication
Summary
User report via freshdesk Failure to login using GitHub authentication
Impact on users
Impacted users unable to access service.
Important information
Here are the steps that reporter has been taken to alleviate their problem. I have tried using both a Mac and Windows OS.
-
Cleared cookies
-
Logged off VPN
-
Tried wifi connections without firewalls
-
Repeated the login process using a different computer
-
Tried using incognito mode
-
Revoking my authorization in GitHub
-
The administrator, Julie ( also tried revoking my authorization
-
Hub URL: http://oppenscapes.2i2c.cloud
-
Support ticket ref: https://2i2c.freshdesk.com/a/tickets/483
Tasks and updates
- Discuss and address incident, leaving comments below with updates
- Incident has been dealt with or is over
- Copy/paste the after-action report below and fill in relevant sections
- Incident title is discoverable and accurate
- All actionable items in report have linked GitHub Issues
After-action report template
# After-action report
These sections should be filled out once we've resolved the incident and know what happened.
They should focus on the knowledge we've gained and any improvements we should take.
## Timeline
_A short list of dates / times and major updates, with links to relevant comments in the issue for more context._
All times in {{ most convenient timezone}}.
- {{ yyyy-mm-dd }} - [Summary of first update](link to comment)
- {{ yyyy-mm-dd }} - [Summary of another update](link to comment)
- {{ yyyy-mm-dd }} - [Summary of final update](link to comment)
## What went wrong
_Things that could have gone better. Ideally these should result in concrete
action items that have GitHub issues created for them and linked to under
Action items._
- Thing one
- Thing two
## Where we got lucky
_These are good things that happened to us but not because we had planned for them._
- Thing one
- Thing two
## Follow-up actions
_Every action item should have a GitHub issue (even a small skeleton of one) attached to it, so these do not get forgotten. These issues don't have to be in `infrastructure/`, they can be in other repositories._
### Process improvements
1. {{ summary }} [link to github issue]
2. {{ summary }} [link to github issue]
### Documentation improvements
1. {{ summary }} [link to github issue]
2. {{ summary }} [link to github issue]
### Technical improvements
1. {{ summary }} [link to github issue]
2. {{ summary }} [link to github issue]
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 39 (37 by maintainers)
my suspicion is that somehow an extra space snuck in @stefaniebutland when you were added and it got suck in random places…
Success!! I’m now logged in. Thanks so much.
For NASA Openscapes Champions, we will collect a list of GitHub usernames and the plan was that @jules32 would add them as she did mine.
Migration to CILogin worked for the user logging in to the hub on chrome.
(i mistakenly pressed reopen on this before from my mobile, closing)
Cleaned up logging to leave cluster in standard state
Waiting for #2277
I’m happy to work with @GeorgianaElena to move from auth0 auth in the openscapes staging and ask the user to retest in Chrome.
I think @GeorgianaElena is working on getting rid of auth0 completely from our infrastructure, which might affect this too. I wonder if we can have this person test using a non-auth0 github integration to see if auth0 is the problem? One of our staging servers perhaps?
I think as we understand the issue is limited to some subset of chrome users with workaround to use another browser we should split off an investigation into understanding if it is an upstream jupyterhub issue, the integration with auth0 to enable use to close the incident. That would allow us to prioritize a more substantial fix alongside the rest of our sprint work.
This should include writing up a support snippet/canned response with a set of steps users can take when get auth failures that users can take to help us diagnose other auth issues
I’m not sure if we need an After-action report or incident write up in this case as it is a bug rather than an incident.
I followed the requests a bit and conclude that the kind of state that the jupyterhub expect is a base64 encoded JSON string starting with
{->ey..., while I saw the samehKstate for the auth0 section.I think the OAuth involved between auth0->github ended up overriding the
statequery parameter of relevance the OAuth involved between jupyterhub->auth0 based on recognizing the state being passed around.This is based on seeing how my successful login request set a cookie state of
ey...and got it returned, while when the 400 surfaced, it was a mismatch betweeney...andhK...as summarized in https://2i2c.slack.com/archives/C028WU9PFBN/p1677263372689249.I logged in as a newly added user
consideratio-demoin an incognito window, and everything worked out. I really don’t understand what is going on.I’d expect any issues with the OAuth2 client’s registered with JupyterHub->Auth0 and Auth0->GitHub should have been erroring systematically if there was any such issue.
Hmm… I visit using
openscapes.2i2c.cloud. Is this possibly an issue related to arriving to the hub using a different domain name or similarly?I doubt it, I’m just throwing out anything I can imagine to cause a “OAuth state mismatch” error which I don’t really understand well.
Having added the roles we now have logs (set to 2w retention for now as this is temp debugging)
Got to cloudwatch in console, log insights, choose the log group
/aws/containerinsights/openscapeshub/applicatioquery language example
Tutorial https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_RunSampleQuery.html
Syntax docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html
Need to add role have done for
eksctl-openscapeshub-nodegroup-co-NodeInstanceRole-1FXJRCFQQ4P4F@GeorgianaElena I think that proposal makes sense as it isn’t impacting all users and we can’t replicate. Given we don’t retain logs and they may be in a TZ where we won’t be around is it worth me adding something that captures the logs and run over the weekend. It’s probably easiest to use AmazonCloudWatch and FluentBit
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-EKS-quickstart.html
The quickstart yaml produced https://gist.github.com/pnasrat/30ad689889118faa8e776d8e88f4b7e4
I was thinking to try to propose them to logout out of everything:
The login into the hub following directly the URL https://openscapes.2i2c.cloud and then check if the 400 shows up again and try to get info out of the logs. Unfortunately I don’t have any smarter ideas 😦 what do you think?
Trying to reproduce using a test GitHub username. Currently unsuccessful. Also, looking at this old issue https://github.com/jupyterhub/jupyterhub/issues/1438
User account appears to be
@jessnicwelchand hub username matches see belowhttps://github.com/jessnicwelch