InvenTree: [BUG] redis.exceptions.ReadOnlyError: You can't write against a read only replica.

Describe the bug

Some reddis replica problem (google don’t help). Reload the containers helps, but it is not an option(

Steps to Reproduce Run docker-compose on production from folder /docker/production After some period of time app on production just crashed.

Expected behavior Just should work

Inventree cache

nventree-cache     | 1:S 04 Jul 2022 19:21:39.896 * Non blocking connect for SYNC fired the event.
inventree-cache     | 1:S 04 Jul 2022 19:21:39.942 # Failed to read response from the server: Connection reset by peer
inventree-cache     | 1:S 04 Jul 2022 19:21:39.943 # Master did not respond to command during SYNC handshake

Iventree worker container inventree-worker | 19:19:54 [Q] ERROR You can't write against a read only replica.

Error log inventree-server container

redis.exceptions.ReadOnlyError: You can't write against a read only replica.
inventree-server    | [04/Jul/2022 14:00:20] "GET /accounts/login/?next=/ HTTP/1.0" 500 59
inventree-server    | [04/Jul/2022 14:00:20] "GET /favicon.ico HTTP/1.0" 302 0
inventree-server    | You can't write against a read only replica.
inventree-server    | Traceback (most recent call last):
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django_redis/cache.py", line 31, in _decorator
inventree-server    |     return method(self, *args, **kwargs)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django_redis/cache.py", line 80, in set
inventree-server    |     return self.client.set(*args, **kwargs)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django_redis/client/default.py", line 185, in set
inventree-server    |     raise ConnectionInterrupted(connection=client) from e
inventree-server    | django_redis.exceptions.ConnectionInterrupted: Redis ReadOnlyError: You can't write against a read only replica.
inventree-server    | 
inventree-server    | During handling of the above exception, another exception occurred:
inventree-server    | 
inventree-server    | Traceback (most recent call last):
inventree-server    |   File "/usr/local/lib/python3.9/wsgiref/handlers.py", line 137, in run
inventree-server    |     self.result = application(self.environ, self.start_response)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/core/handlers/wsgi.py", line 133, in __call__
inventree-server    |     response = self.get_response(request)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/core/handlers/base.py", line 130, in get_response
inventree-server    |     response = self._middleware_chain(request)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/core/handlers/exception.py", line 49, in inner
inventree-server    |     response = response_for_exception(request, exc)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/core/handlers/exception.py", line 114, in response_for_exception
inventree-server    |     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/core/handlers/exception.py", line 153, in handle_uncaught_exception
inventree-server    |     return callback(request)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
inventree-server    |     response = view_func(request, *args, **kwargs)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/views/defaults.py", line 97, in server_error
inventree-server    |     return HttpResponseServerError(template.render())
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/template/backends/django.py", line 61, in render
inventree-server    |     return self.template.render(context)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/template/base.py", line 170, in render
inventree-server    |     return self._render(context)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
inventree-server    |     return self.nodelist.render(context)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/template/base.py", line 938, in render
inventree-server    |     bit = node.render_annotated(context)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
inventree-server    |     return self.render(context)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/template/loader_tags.py", line 150, in render
inventree-server    |     return compiled_parent._render(context)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
inventree-server    |     return self.nodelist.render(context)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/template/base.py", line 938, in render
inventree-server    |     bit = node.render_annotated(context)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
inventree-server    |     return self.render(context)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django/template/library.py", line 192, in render
inventree-server    |     output = self.func(*resolved_args, **resolved_kwargs)
inventree-server    |   File "/home/ubuntu/InvenTree/InvenTree/part/templatetags/inventree_extras.py", line 300, in settings_value
inventree-server    |     return InvenTreeSetting.get_setting(key)
inventree-server    |   File "/home/ubuntu/InvenTree/InvenTree/common/models.py", line 367, in get_setting
inventree-server    |     setting = cls.get_setting_object(key, **kwargs)
inventree-server    |   File "/home/ubuntu/InvenTree/InvenTree/common/models.py", line 353, in get_setting_object
inventree-server    |     setting.save_to_cache()
inventree-server    |   File "/home/ubuntu/InvenTree/InvenTree/common/models.py", line 98, in save_to_cache
inventree-server    |     cache.set(
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django_redis/cache.py", line 38, in _decorator
inventree-server    |     raise e.__cause__
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/django_redis/client/default.py", line 175, in set
inventree-server    |     return bool(client.set(nkey, nvalue, nx=nx, px=timeout, xx=xx))
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/redis/commands/core.py", line 2220, in set
inventree-server    |     return self.execute_command("SET", *pieces, **options)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/redis/client.py", line 1238, in execute_command
inventree-server    |     return conn.retry.call_with_retry(
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
inventree-server    |     return do()
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/redis/client.py", line 1239, in <lambda>
inventree-server    |     lambda: self._send_command_parse_response(
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/redis/client.py", line 1215, in _send_command_parse_response
inventree-server    |     return self.parse_response(conn, command_name, **options)
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/redis/client.py", line 1254, in parse_response
inventree-server    |     response = connection.read_response()
inventree-server    |   File "/home/ubuntu/InvenTree/dev/env/lib/python3.9/site-packages/redis/connection.py", line 839, in read_response
inventree-server    |     raise response
inventree-server    | redis.exceptions.ReadOnlyError: You can't write against a read only replica.
inventree-server    | [04/Jul/2022 14:00:20] "GET /accounts/login/?next=/favicon.ico HTTP/1.0" 500 59

Deployment Method

• Docker

Version Information Latest stable branch