infrastructure: [Incident] Kernel restarts and internal gateway errors for for U. Texas hub

Summary

We recently deployed the UT Austin hub, which is running a database sidecar for each student (https://github.com/2i2c-org/infrastructure/issues/968). Since the hub began use, we’ve gotten reports of intermittent kernel crashes affecting students. This has been difficult to reproduce, but has been happening consistently.

There was originally a FreshDesk ticket about this here: https://2i2c.freshdesk.com/a/tickets/84?lang=en

We thought we had resolved it by increasing the memory limits for each user session, but these issues have persisted. Here are the two major things that we’ve found:

  1. Users experiencing “kernel died” errors sporadically
  2. Users getting “internal gateway errors” when using pgadmin (https://github.com/2i2c-org/infrastructure/issues/968#issuecomment-1048229587)

Here’s an issue where we describe some of the problems people have, and how to reproduce them:

https://github.com/2i2c-org/infrastructure/pull/1008#issuecomment-1047372853

Hub information

Timeline (if relevant)

All times in US/Pacific.

2022-02-17

  • @jameshowison reports that student kernels are dying via FreshDesk

2022-02-18

We conclude that this may be a problem with memory limits students are hitting during queries, and bump their guarantee here: https://github.com/2i2c-org/infrastructure/pull/1008

2022-02-21

We get reports that many students are still running into this problem. It is not reliable to replicate, but people are consistently hitting these errors.

ref: https://github.com/2i2c-org/infrastructure/pull/1008#issuecomment-1047040355

TODO: add more as new developments happen


After-action report

These sections should be filled out once we’ve resolved the incident and know what happened. They should focus on the knowledge we’ve gained and any improvements we should take.

What went wrong

Things that could have gone better. Ideally these should result in concrete action items that have GitHub issues created for them and linked to under Action items.

Where we got lucky

These are good things that happened to us but not because we had planned for them.

Action items

These are only sample subheadings. Every action item should have a GitHub issue (even a small skeleton of one) attached to it, so these do not get forgotten. These issues don’t have to be in infrastructure/, they can be in other repositories.

Process improvements

  1. {{ summary }} [link to github issue]
  2. {{ summary }} [link to github issue]

Documentation improvements

  1. {{ summary }} [link to github issue]
  2. {{ summary }} [link to github issue]

Technical improvements

  1. {{ summary }} [link to github issue]
  2. {{ summary }} [link to github issue]

Actions

  • Incident has been dealt with or is over
  • Sections above are filled out
  • Incident title and after-action report is cleaned up
  • All actionable items above have linked GitHub Issues

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 37 (22 by maintainers)

Commits related to this issue

Most upvoted comments

No sign of kernel issues during class today. No repeat of the “Invalid Response Error”. I will report as student do their assignments remainder of the week and weekend.

I think current state is:

  1. xeus-sql is producing some weird 504 gateway errors
  2. ipython-sql works alright, doesn’t cause any issues
  3. xeus-sql produces results sometimes that are too large and freezes user browsers
  4. pgadmin4 is still broken

Is this an accurate summary of current status, @jameshowison? If ipython-sql works fine, then perhaps a longer term outlook is to move towards making sure ipython-sql works well, and fix pgadmin4…

Only me working with it at present, but I’m working in ipython rather than xues-sql, so I have a backup if it does.

This is really excellent investigative engineering @yuvipanda 😃

Would it be helpful for us to document the jupyterlab-limit-output extension as well? I’m happy to submit a PR if we think it’ll be common-enough to put in docs/

(as an aside, what a strange world we live in where we can just submit a patch to a random JupyterLab extension developed by DEShaw)

I’ve also opened https://github.com/deshaw/jupyterlab-limit-output/pull/2 to add suppoort for xeus-sql in jupyterlab-limit-output

There are basically three issues:

  1. Kernel dying
  2. Browser freezing
  3. pgadmin4 not working

Today, I’ve put in place fixes for (1) and (2), but not (3). Intermittent issues are always a bit of a mess to debug… I’m hopeful I can put in some more time for (3) tomorrow.

@jameshowison I’ve brought in a newer version of xeus-sql with https://github.com/jupyter-xeus/xeus-sql/issues/60 merged, so hopefully the error logs are more helpful next time.

On the postgresql container side, I can see that there’s an error message saying ‘FATAL: terminating connection due to administrator command’ (query), and its frequence looks like this:

image

Looking around, it looks like this might becaused by external process sending SIGTERM to the database (https://www.postgresql.org/message-id/4564.1284559661@sss.pgh.pa.us). In our case, that might be because postgres is hitting a memory limit. I’m going to bump the limit up and see if that helps - earlier we had just bumped up the limit for the data the kernel can handle, but now am going to do that for the database itself.

I’m sorry you’re running into this, @jameshowison - definitely frustrating. Thank you for helping us debug as we go along!

Notebook logs are written to ~/.jupyter-server-log.txt file - you can check that out. I looked and found that xeus-sql was crashing with an error message terminate called after throwing an instance of 'char const*'. I used our Google Cloud logger (query) to search for that particular error message in all notebook logs, and found that has been happening for a while (just looking at last 7 days)

image

I think if we could actually get the value of the char const* we might be able to get a better idea of what the error is. I’ve opened https://github.com/jupyter-xeus/xeus-sql/issues/60 to let the xeus-sql developers know.