nautobot: SSL error: decryption failed or bad record mac & SSL SYSCALL error: EOF detected

Environment

  • Python version: 3.8.5
  • Nautobot version: 1.0.0b2

Steps to Reproduce

  1. become user nautobot
  2. Download & extract 1.0.0b2
  3. Execute install.ssh (use existing config except unicorn.py; Adapt changes in systemd service files)
  4. Start nautobot
  5. Open Nautobot WebUI and surf a bit

This only happens after a fresh start of nautobot. The error disappears after a while until a nautobot restart.

Expected Behavior

no HTTP500

Observed Behavior

fyi: /opt/nautobot/current is a symlink to /opt/nautobot/1.0.0b2/

Server Error
There was a problem with your request. Please contact an administrator.

The complete exception is provided below:

<class 'django.db.utils.OperationalError'>

SSL error: decryption failed or bad record mac

Python version: 3.8.5
Nautobot version: 1.0.0b2


OperationalError at /plugins/device-onboarding/
SSL error: decryption failed or bad record mac
Request Method:	GET
Request URL:	https://<schnipp>/plugins/device-onboarding/
Django Version:	3.1.7
Exception Type:	OperationalError
Exception Value:	
SSL error: decryption failed or bad record mac
Exception Location:	/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py, line 84, in _execute
Python Executable:	/usr/bin/python3.8
Python Version:	3.8.5
Python Path:	
['.',
 '',
 '/opt/nautobot/1.0.0b2/venv/bin',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/git/ext/gitdb',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/gitdb/ext/smmap']
Server time:	Wed, 10 Mar 2021 12:34:59 +0000

Environment:


Request Method: GET
Request URL: https://<schnipp>/plugins/device-onboarding/

Django Version: 3.1.7
Python Version: 3.8.5
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'cacheops',
 'corsheaders',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'mptt',
 'rest_framework',
 'taggit',
 'timezone_field',
 'nautobot.core',
 'nautobot.circuits',
 'nautobot.dcim',
 'nautobot.ipam',
 'nautobot.extras',
 'nautobot.tenancy',
 'nautobot.users',
 'nautobot.utilities',
 'nautobot.virtualization',
 'django_rq',
 'drf_yasg',
 'graphene_django',
 'nautobot_device_onboarding.OnboardingConfig']
Installed Middleware:
['django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'nautobot.core.middleware.ExceptionHandlingMiddleware',
 'nautobot.core.middleware.RemoteUserMiddleware',
 'nautobot.core.middleware.APIVersionMiddleware',
 'nautobot.core.middleware.ObjectChangeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']



Traceback (most recent call last):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 233, in _get_session
    return self._session_cache

During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

The above exception (SSL error: decryption failed or bad record mac
) was the direct cause of the following exception:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/utilities/views.py", line 91, in dispatch
    if not self.has_permission():
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/utilities/views.py", line 73, in has_permission
    if user.has_perms((permission_required, *self.additional_permissions)):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 240, in inner
    self._setup()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 376, in _setup
    self._wrapped = self._setupfunc()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 23, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 11, in get_user
    request._cached_user = auth.get_user(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 174, in get_user
    user_id = _get_user_session_key(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 58, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 65, in __getitem__
    return self._session[key]
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 238, in _get_session
    self._session_cache = self.load()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 43, in load
    s = self._get_session_from_db()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 32, in _get_session_from_db
    return self.model.objects.get(
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 425, in get
    num = len(clone)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 273, in _fetch_all
    return self._no_monkey._fetch_all(self)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

Exception Type: OperationalError at /plugins/device-onboarding/
Exception Value: SSL error: decryption failed or bad record mac

Seconde error:

OperationalError at /
SSL SYSCALL error: EOF detected
Request Method:	GET
Request URL:	https://<schnipp>/
Django Version:	3.1.7
Exception Type:	OperationalError
Exception Value:	
SSL SYSCALL error: EOF detected
Exception Location:	/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py, line 84, in _execute
Python Executable:	/usr/bin/python3.8
Python Version:	3.8.5
Python Path:	
['.',
 '',
 '/opt/nautobot/1.0.0b2/venv/bin',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/git/ext/gitdb',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/gitdb/ext/smmap']
Server time:	Wed, 10 Mar 2021 13:06:33 +0000

Environment:


Request Method: GET
Request URL: https://<schnipp>/

Django Version: 3.1.7
Python Version: 3.8.5
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'cacheops',
 'corsheaders',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'mptt',
 'rest_framework',
 'taggit',
 'timezone_field',
 'nautobot.core',
 'nautobot.circuits',
 'nautobot.dcim',
 'nautobot.ipam',
 'nautobot.extras',
 'nautobot.tenancy',
 'nautobot.users',
 'nautobot.utilities',
 'nautobot.virtualization',
 'django_rq',
 'drf_yasg',
 'graphene_django',
 'nautobot_device_onboarding.OnboardingConfig']
Installed Middleware:
['django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'nautobot.core.middleware.ExceptionHandlingMiddleware',
 'nautobot.core.middleware.RemoteUserMiddleware',
 'nautobot.core.middleware.APIVersionMiddleware',
 'nautobot.core.middleware.ObjectChangeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']



Traceback (most recent call last):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 233, in _get_session
    return self._session_cache

During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

The above exception (SSL SYSCALL error: EOF detected
) was the direct cause of the following exception:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/core/views/__init__.py", line 45, in get
    ConsolePort.objects.restrict(request.user, "view")
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/utilities/querysets.py", line 21, in restrict
    if user.is_superuser or permission_is_exempt(permission_required):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 240, in inner
    self._setup()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 376, in _setup
    self._wrapped = self._setupfunc()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 23, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 11, in get_user
    request._cached_user = auth.get_user(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 174, in get_user
    user_id = _get_user_session_key(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 58, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 65, in __getitem__
    return self._session[key]
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 238, in _get_session
    self._session_cache = self.load()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 43, in load
    s = self._get_session_from_db()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 32, in _get_session_from_db
    return self.model.objects.get(
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 425, in get
    num = len(clone)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 273, in _fetch_all
    return self._no_monkey._fetch_all(self)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

Exception Type: OperationalError at /
Exception Value: SSL SYSCALL error: EOF detected

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 23 (16 by maintainers)

Most upvoted comments

This only affects Ubuntu systems where PostgreSQL was installed via apt, in which SSL is enabled by default. You can confirm this by connecting to the database using psql and running \conninfo:

$ psql nautobot -U nautobot -h localhost --password
Password:
psql (12.6 (Ubuntu 12.6-0ubuntu0.20.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

nautobot=> \conninfo
You are connected to database "nautobot" as user "nautobot" on host "localhost" (address "::1") at port "5432".
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)

After more debugging, I can 100% get the issue to go away if I do any one of these:

  • Don’t use processes = in uwsgi.ini or --processes CLI flag when starting uWSGI, which doesn’t result in pre-forking
  • Disable SSL connection to PGSQL by setting DATABASES['default']['OPTIONS'] = {"sslmode": "disable"} in nautobot_config.py
  • Disable persistent connections by setting DATABASES['default']['CONN_MAX_AGE'] = 0 in nautobot_config.py, which is the Django default to always close the connection on every request.

Obviously the first two are not desirable for production use. The third option in my opinion should be our new default for the common case if we’re going to continue to recommend uWSGI over Gunicorn.

The root cause is the aggressive pre-forking of uWSGI which performs copy-on-write on subprocesses that often duplicates the database connection objects. This issue did not arise when using Gunicorn because its default sync worker class defers to every worker handling only a single request before cycling and deliberately closing the database connection. This mirrors the default behavior of Django with CONN_MAX_AGE=0, which is why when I set that, uWSGI behaves as expected.

+2
jathanism on Mar 13, 2021

@mkrsn and @jathanism, thanks for having this conversation. I’ve run into the same problem (using the new install instructions, on Ubutntu 20.04, using the distro-supplied PostgreSQL), and have worked around it by setting the database MAX_CONN_AGE to zero.

+1
chrismarget on Mar 15, 2021

Confirmed - disabling the conn-encryption to postgres solves the issue.

… If you don’t mind, I’ll use this as inspiration for the forthcoming documentation changes.

sure

+1
mkrsn on Mar 15, 2021

Nice work, thanks 😃 I’ll try to disable Postgres encryption on Monday and report back.

But wouldn’t it be better to file a Bug at the uWSGI project instead of defining a workaround as new default?

If you really want to change the default config to workaround that problem maybe it would be better to let the user choose the best solution. This would just need a small note in the Documentation.

In a all-in-one Setup (bare metal & VM, not LXC, LXD or Docker), while just talking to localhost, the best solution would imho be to disable the encryption to the Postgres because:

  • You need priv rights (or must be in the pcap group - which is also root decision) to sniff Network traffic
  • If an attacker gets root - you have bigger Problems than a plain connection to localhost
  • Persistence DB Connections increase the Performance alot (with activated TLS even more (3-Way TCP handshake, TLS Handshake & 4-Way TCP teardown)) and decreases the Time to first Byte.
+1
mkrsn on Mar 13, 2021

@mkrsn That’s great feedback, thanks! I’d like to get to the bottom of the root cause to this other than just trying to circumvent it. This helps narrow it down.

+1
jathanism on Mar 12, 2021

Type: VM (kvm) OS: Ubuntu 20.04.2 LTS Postgres: 12.6-0ubuntu0.20.04.1 Redis: 5:5.0.7-2 RAM: 8GiB (~1GiB used; ~3GiB cached; ~4GiB free) vCPU: 4

I’ve already switched from gunicorn to uWSGI. I’ll get back on you after installing nautobot via new install instructions.

fyi: Our nautobot database is currently populated with a handful of demo data (just 4 devices, some IPs, a Rack and so on).

+1
mkrsn on Mar 10, 2021
Read more comments on GitHub
← nautobot: pynautobot reqeust error
nautobot: Unable to stand up nautobot in a HA mode (one write instance & one read-only instance) →