django-post_office: SMTPServerDisconnected error with send_mail and Celery

Try just up Python version

Like @cberzan wrote, it’s because of connection cache and smtp timeouts. I have some dirty hack that fixes this issue for me. I’m using it with django.core.mail.backends.smtp.EmailBackend

from post_office.connections import connections

connections._connections.connections['default'].close()  # because it might be opened and we can't check
connections._connections.connections['default'].open()
# actual code for sending messages
...

connections._connections.connections['default'].close()

Proper solution is IMO add option to disable connection caching when using long running threads.

Seems that the Email.dispatch() method used to close the connection, but that changed in commit 03d1a37 and now the connection times out.

I changed it later, works on production without problems:

try:
    connections._connections.connections['default'].close()
    connections._connections.connections['default'].open()
except AttributeError:
    pass

Don’t judge me 😃

I will try to find some time to make pull request with disabling connection caching based on settings. It will solve all of the problems.

Don’t use priority NOW, especially not together with Celery. It thwards the main reason for using post_office, namely its asynchronously. With Celery, emails are sent immediately after they have been queued, but by another task.

How about just close the connection after sending email with priority NOW ?

As in https://github.com/haiwen/django-post_office/commit/2312cf240363721f737b5ac8eb86ab8cb255938f

@VerosK we use EMAIL_POST_OFFICE_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' and Microsoft Office.

Current set up is using send_mail without celery task and created scheduled Celery task every minute like this:

@app.task(name='send_queued_mail')
def send_queued_mail():
    call_command('send_queued_mail')

with following settings:

POST_OFFICE = {
    'DEFAULT_PRIORITY': 'high'
}

we also have following celery task to send failed emails every 2 minutes:

@app.task(name='send_failed_mails')
def send_failed_mails():
    from post_office.models import Email, STATUS, Log

    max_retries = getattr(settings, 'EMAIL_MAX_RETRIES', 5)
    # select FAILED email WHERE num_logs <= max
    query_set = Email.objects.annotate(num_logs=Count('logs')).filter(Q(status__isnull=True) | Q(status=STATUS.failed)).filter(num_logs__lte=max_retries).all()
    logs = []
    for mail in query_set:
        if not mail.status:
            logger.error('Email without status with id: {} from: {}, to: {}'.format(mail.id, mail.from_email, mail.to))
        if mail.logs.count() == max_retries:
            logger.error('Unable to send mail with id: {} from: {}, to: {}'.format(mail.id, mail.from_email, mail.to))
            # here we create new log to stop sending over and over
            logs.append(Log(email=mail, status=STATUS.failed, message='maximum retries exceeded', exception_type='maximum retries exceeded'))
        else:
            mail.status = STATUS.queued
        mail.save()
    Log.objects.bulk_create(logs)

I also facing to issue with 'DEFAULT_PRIORITY': 'now' some emails ends up with status None. Reason was probably that when you don’t have setting EMAIL_TIMEOUT, email is created but it takes to long and wsgi kill process and you end with email with status None.