kombu: "Error 110 while writing to socket. Connection timed out." With kombu 4.4.0/4.5.0 and redis 3.2.0/3.2.1

I was previously on kombu 4.3.0 and redis 2.10.6, and after upgrading to kombu 4.4.0/4.5.0 and redis 3.2.0/3.2.1 I noticed a new issue on my Django REST endpoints:

Error 110 while writing to socket. Connection timed out.

These endpoints never had any issues before and this issue popped up only once since I upgraded, but happened on both endpoints simultaneously

Edit: Bug still there in most recent versions of the libraries (celery 4.3.0, kombu 4.5.0, redis 3.2.1) (I also had the bug when just upgrading kombu/redis and keeping celery at 4.2.2) Bug is not there in celery 4.2.2, kombu 4.3.0, redis 2.10.6

2019-05-21T07:51:02.022118+00:00 app[worker.1]: [2019-05-21 07:51:02,021: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (0/20) now.
2019-05-21T07:51:02.024750+00:00 app[worker.1]: [2019-05-21 07:51:02,024: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (1/20) in 1.00 second.
2019-05-21T07:51:03.028332+00:00 app[worker.1]: [2019-05-21 07:51:03,028: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (2/20) in 1.00 second.
2019-05-21T07:51:04.032513+00:00 app[worker.1]: [2019-05-21 07:51:04,032: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (3/20) in 1.00 second.
2019-05-21T07:51:05.037741+00:00 app[worker.1]: [2019-05-21 07:51:05,037: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (4/20) in 1.00 second.
2019-05-21T07:51:06.041513+00:00 app[worker.1]: [2019-05-21 07:51:06,041: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (5/20) in 1.00 second.
2019-05-21T07:51:07.045367+00:00 app[worker.1]: [2019-05-21 07:51:07,045: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (6/20) in 1.00 second.
2019-05-21T07:51:08.048339+00:00 app[worker.1]: [2019-05-21 07:51:08,048: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (7/20) in 1.00 second.
2019-05-21T07:51:09.052390+00:00 app[worker.1]: [2019-05-21 07:51:09,052: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (8/20) in 1.00 second.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 53 (18 by maintainers)

Commits related to this issue

Use AWS SQS broker and Django DB result backend with Celery (#821) In #814 I added optional Celery integration with Redis as the broker, but this proved to be unreliable due to celery/kombu#1019. The... — committed to JustFixNYC/tenants2 by toolness 5 years ago
Issue #1019 Fix redis transport socket timeout Patch from @jschwartzentruber https://github.com/celery/kombu/issues/1019#issuecomment-535157400 — committed to kmctown/kombu by kmctown 5 years ago
Issue #1019 Fix redis transport socket timeout Patch from @jschwartzentruber https://github.com/celery/kombu/issues/1019#issuecomment-535157400 — committed to kmctown/kombu by kmctown 5 years ago
Issue #1019 Fix redis transport socket timeout Patch from @jschwartzentruber https://github.com/celery/kombu/issues/1019#issuecomment-535157400 — committed to kmctown/kombu by kmctown 5 years ago
Issue #1019 Fix redis transport socket timeout (#1113) * Issue #1019 Fix redis transport socket timeout Patch from @jschwartzentruber https://github.com/celery/kombu/issues/1019#issuecomment-53515... — committed to celery/kombu by kmctown 5 years ago
configure celery broker to health check redis this might solve one of the issues we see when our redis pod restarts https://github.com/celery/kombu/issues/1019 — committed to alexnsu/lemur by alexnsu 2 years ago
Add a health check from celery to redis (#212) * Add a health check from celery to redis this solved connection problems between celery and redis for other peeps https://github.com/celery/kombu/is... — committed to spotify/lemur by deleted user 2 years ago

Most upvoted comments

I’m unfortunately in no position to fix this, but I thought I’d at least make sure that the dots were connected between redis-py and here. Over on their upstream issue, they just did a release (3.3.0) with a new feature, and they say:

For Celery users, this change won’t automatically fix ConnectionErrors encountered by Celery. Celery uses PubSub in a non-standard way which can not take advantage of the automatic health checks at this time. Once this code is released, we should be able to create a PR for Celery to regularly call pubsub.check_health().

(https://github.com/andymccurdy/redis-py/issues/1140#issuecomment-515634842)

It’d be great to have Celery start using this functionality.

mlissner on Aug 5, 2019

Right, for anyone reading this, it seems the issue appeared from the 3.x branch of redispy indeed (which is required by celery 4.3+ / kombu 4.4+), and the status of the issue is evolving there:

https://github.com/andymccurdy/redis-py/issues/1140

alexandre-paroissien on Jun 20, 2019

After upgrading we haven’t seen the connection time out issue since.

@hampsterx what package and version did you upgrade to?

westonplatter on May 27, 2021

what version of redis are you running? we had a version prior to 5.x and had this issue with following libs noted at the time:

redis 3.2.0
celery 4.3.0

After upgrading we haven’t seen the connection time out issue since.

Worth ruling out redis in any case…

hampsterx on May 13, 2021

I am not using eventlet

I just posted this comment on redis-py https://github.com/andymccurdy/redis-py/issues/1140#issuecomment-472712115

alexandre-paroissien on Mar 14, 2019