infrastructure: [Incident] OceanHackWeek hub cannot start server
Summary
OceanHackWeek hub cannot start a server. Reported in https://2i2c.freshdesk.com/a/tickets/172
Impact on users
Hub unusable as no one can start a server.
Important information
- Hub URL: oceanhackweek.2i2c.cloud
- Support ticket ref: https://2i2c.freshdesk.com/a/tickets/172
Tasks and updates
- Discuss and address incident, leaving comments below with updates
- Incident has been dealt with or is over
- Copy/paste the after-action report below and fill in relevant sections
- Incident title is discoverable and accurate
- All actionable items in report have linked GitHub Issues
After-action report template
# After-action report
These sections should be filled out once we've resolved the incident and know what happened.
They should focus on the knowledge we've gained and any improvements we should take.
## Timeline
_A short list of dates / times and major updates, with links to relevant comments in the issue for more context._
All times in {{ most convenient timezone}}.
- {{ yyyy-mm-dd }} - [Summary of first update](link to comment)
- {{ yyyy-mm-dd }} - [Summary of another update](link to comment)
- {{ yyyy-mm-dd }} - [Summary of final update](link to comment)
## What went wrong
_Things that could have gone better. Ideally these should result in concrete
action items that have GitHub issues created for them and linked to under
Action items._
- Thing one
- Thing two
## Where we got lucky
_These are good things that happened to us but not because we had planned for them._
- Thing one
- Thing two
## Follow-up actions
_Every action item should have a GitHub issue (even a small skeleton of one) attached to it, so these do not get forgotten. These issues don't have to be in `infrastructure/`, they can be in other repositories._
### Process improvements
1. {{ summary }} [link to github issue]
2. {{ summary }} [link to github issue]
### Documentation improvements
1. {{ summary }} [link to github issue]
2. {{ summary }} [link to github issue]
### Technical improvements
1. {{ summary }} [link to github issue]
2. {{ summary }} [link to github issue]
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 21 (21 by maintainers)
We just (~12:26 Eastern) had the hub lock up again, but things seemed to recover immediately after hitting the ‘Shutdown Hub’ button.
I am going to close this issue now because:
Yes, that’s correct!
I found these logs in another hub that had the same symptoms at the same time:
This is definitely due to the GKE master having a hiccup. It usually recovers shortly, but it looks like the jupyterhub process isn’t 😦 We should definitely report this upstream.
@abkfenris We suspect this is an issue with the availability of the k8s master. We are seeing spawning processes being cancelled, should not be related to the NFS. The 2i2c cluster is not regional, so it does not have high availability of the k8s master so we see issues like this occasionally and the fix is to restart the hub pod. We have an issue to move the cluster to be a regional one, but it would be a destructive process and we need to coordinate appropriate downtime with everyone who has a hub running on this cluster: https://github.com/2i2c-org/infrastructure/issues/1102
Yuvi has just informed me that we do indeed have logs in the GCP console! They’re a bit hard to read though, but there are a lot of
CancelledErrorsstemming from asyncio from before the incident report.