channels_redis: ConnectionClosedError when discarding old channels

Summary

I’m seeing a lot of the following errors in Sentry:

ConnectionClosedError: Reader at end of file
  File "django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 126, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 124, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 483, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 443, in handle_exception
    self.raise_uncaught_exception(exc)
  File "rest_framework/views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "api/views.py", line 411, in put
    return self.update(request, *args, **kwargs)
  File "rest_framework/mixins.py", line 70, in update
    self.perform_update(serializer)
  File "rest_framework/mixins.py", line 80, in perform_update
    serializer.save()
  File "rest_framework/serializers.py", line 209, in save
    self.instance = self.update(self.instance, validated_data)
  File "api/serializers.py", line 227, in update
    'screen_shot': nami_state_instance.state_favorite
  File "api/serializers.py", line 104, in send_realtime_update
    'type': 'session_message',
  File "asgiref/sync.py", line 64, in __call__
    return call_result.result()
  File "concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "asgiref/sync.py", line 78, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "channels_redis/core.py", line 561, in group_send
    await connection.zremrangebyscore(key, min=0, max=int(time.time()) - self.group_expiry)
  File "aioredis/commands/sorted_set.py", line 268, in zremrangebyscore
    return self.execute(b'ZREMRANGEBYSCORE', key, min, max)
  File "aioredis/commands/__init__.py", line 50, in execute
    return self._pool_or_conn.execute(command, *args, **kwargs)
  File "aioredis/connection.py", line 287, in execute
    raise ConnectionClosedError(msg)

Steps to Reproduce

  • Deploy django channels on Heroku with the minimum infra (1 dyno, 1 paid instance of redis).
  • Create an endpoint that produces lots of events that can be consumed elsewhere.
  • Create a consumer that consumes these events.
  • Ping that endpoint and “produce” tons of events using async_to_sync(channel_layer.group_send).
  • Subscribe to these events from the consumer in your browser.
  • Observe the error described above.

Expected Results

This error is a bit vague so it would be helpful if we could point to some likely root causes for why the connection is closed.

It would also be helpful to list all common exceptions that can occur when using group_send.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 27 (5 by maintainers)

Most upvoted comments

@lyubo1 I think you are right; this error seems to only happen when I have a page open longer than 5 minutes.

See this issue which I think is the same: https://github.com/django/channels_redis/issues/164

As I wrote there, I tried the solution you suggested, but still encounter the same problem. Anyway, it seems this issue isn’t limited to Heroku but is rather related to the timeout setting as you said.