django-health-check: Django sending Internal Server Error mails for health check endpoint

After upgrading our Django app (Django 2.0.9 -> 2.1.3, Python 3.6 -> 3.7, django-health-check 3.5.1 -> 3.8.0) we’re sometimes getting Internal Server Error mails from Django triggered by the health check endpoint. Unfortunately these mails do not contain a stack trace or any other information that is helpful in determining what’s going on, which is why I’m opening this issue.

We’re running our app on Kubernetes, and it is configured to periodically call the health check endpoint to verify that the app is still alive. There are django-health-check checks configured for db (PostgreSQL) and cache (memcached).

The first part of the error mail is as follows:

Internal Server Error: /healthz

Report at /healthz
Internal Server Error: /healthz

Request Method: GET
Request URL: http://10.244.2.193:8000/healthz
Django Version: 2.1.3
Python Executable: /usr/local/bin/python
Python Version: 3.7.1
Python Path: ['/srv', '/usr/local/bin', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/site-packages']
Server time: vr, 9 Nov 2018 19:14:47 +0100
Installed Applications:
(...
 'health_check',
 'health_check.db',
 'health_check.cache',
...)

Do you happen to have any idea what could be causing these internal server errors?

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 18

Most upvoted comments

Cheers! Enabling the health-check logging did the trick. Apparently the Internal Server Errors were caused by a failing cache health check:

2018-11-18 10:46:15,673 ERROR health-check.add_error:49 unavailable: Cache key does not match
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/usr/local/lib/python3.7/site-packages/health_check/cache/backends.py", line 14, in check_status
    raise ServiceUnavailable("Cache key does not match")
health_check.exceptions.ServiceUnavailable: unavailable: Cache key does not match
2018-11-18 10:46:15,702 ERROR django.request.log_response:228 Internal Server Error: /healthz

I’m not quite sure why this happens (memcached is running alongside Django, so it cannot be a network problem), but at least there’s some context.

praseodym on Nov 18, 2018

Hi @GitRon sure, it’d be my pleasure.

By the health-check-logger, I mean the name of the logger. In Django you can configure the handlers for each logger or other behavior via the LOGGING setting.

In this particular case you want to add:

LOGGING = {
    # ...
    'loggers': {
        # ...
        'health-check': {
            'handlers': ['console'],
            'level': 'WARNING',
            'propagate': True,
        }
}

See also Django’s documentation on logging for more information: https://docs.djangoproject.com/en/stable/topics/logging/

codingjoe on Jun 15, 2019