docker-mailserver: bug report: Dovecot compression not working

๐Ÿ“ Preliminary Checks

  • I tried searching for an existing issue and followed the debugging docs advice, but still need assistance.

๐Ÿ‘€ What Happened?

I have the following setting in dovecot.cf

protocol imap {
	mail_plugins = $mail_plugins zlib imap_zlib
}

# Enable these only if you want compression while saving:
plugin {
	zlib_save = gz
	zlib_save_level = 9
}

plugin {
	## NOTE! Setting was called imap_zlib_compression_level before 2.3.15
	## imap_compress_<algorithm>_level = value
	##
	imap_compress_deflate_level = 6
}

๐Ÿ‘Ÿ Reproduction Steps

New mail that arrives in maildir is does not get compressed. These settings should save all new email compressed with gz.

๐Ÿ‹ DMS Version

v14.0.0

๐Ÿ’ป Operating System and Architecture

Debian 11 x86_64

โš™๏ธ Container configuration files

services:
  mailserver:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    read_only: false
    container_name: mailserver
    # Provide the FQDN of your mail server here (Your DNS MX record should point to this value)
    hostname: mail.emimx.com
    environment:
      - SSL_TYPE=manual
      - SSL_CERT_PATH=/tmp/dms/custom-certs/fullchain.cer
      - SSL_KEY_PATH=/tmp/dms/custom-certs/cert.key   
    env_file: mailserver.env
    # More information about the mail-server ports:
    # https://docker-mailserver.github.io/docker-mailserver/latest/config/security/understanding-the-ports/
    ports:
      - "25:25"    # SMTP  (explicit TLS => STARTTLS, Authentication is DISABLED => use port 465/587 instead)
      - "143:143"  # IMAP4 (explicit TLS => STARTTLS)
      - "465:465"  # ESMTP (implicit TLS)
      - "587:587"  # ESMTP (explicit TLS => STARTTLS)
      - "993:993"  # IMAP4 (implicit TLS)
    volumes:
      - ./mail/:/var/mail/
      - ./mail-state/:/var/mail-state/
      - ./mail-logs/:/var/log/mail/
      - ./config/:/tmp/docker-mailserver/
      - ./letsencrypt:/etc/letsencrypt
      - /etc/localtime:/etc/localtime:ro
      - /etc/ssl/private/cert.ecc/:/tmp/dms/custom-certs/:ro
    restart: unless-stopped
    stop_grace_period: 1m
    # Uncomment if using `ENABLE_FAIL2BAN=1`:
    # cap_add:
    #   - NET_ADMIN
    healthcheck:
      test: "ss --listening --tcp | grep -P 'LISTEN.+:smtp' || exit 1"
      timeout: 3s
      retries: 0

๐Ÿ“œ Relevant log output

new email stored uncompressed

Improvements to this form?

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (11 by maintainers)

Most upvoted comments

@polarathene Wow, thank you very much for this very thorough help. I really, really appreciate that! Apologies for the late reply, I was travelling. Based on the evidence you provided, I confirm tha.t this was a mis-configuration on my part.

However, you seem to actually need zlib in protocol lmtp and protocol imap. Using your configuration the email is written compressed but the imap plugin than reports a stream error because the size of the email is different from the size in the name and therefore is not read.

In the end I simply modified my dovecot.cf to to include compression. In case you are interested (or anyone else reading this) here is the configuration that works for me: dovecot.cf

login_trusted_networks = 127.0.0.1

protocol lmtp {
	mail_plugins = $mail_plugins zlib
}

protocol imap {
	mail_plugins = $mail_plugins zlib imap_zlib
}

# Enable these only if you want compression while saving:
plugin {
	zlib_save = gz
	zlib_save_level = 9
}

plugin {
	imap_compress_deflate_level = 6
}

And here is my compose.yaml

services:
  mailserver:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    read_only: false
    container_name: mailserver
    # Provide the FQDN of your mail server here (Your DNS MX record should point to this value)
    hostname: deleted
    environment:
      - SSL_TYPE=manual
      - SSL_CERT_PATH=deleted
      - SSL_KEY_PATHdeleted
    env_file: mailserver.env
    # More information about the mail-server ports:
    # https://docker-mailserver.github.io/docker-mailserver/latest/config/security/understanding-the-ports/
    ports:
      - "25:25"    # SMTP  (explicit TLS => STARTTLS, Authentication is DISABLED => use port 465/587 instead)
      - "143:143"  # IMAP4 (explicit TLS => STARTTLS)
      - "465:465"  # ESMTP (implicit TLS)
      - "587:587"  # ESMTP (explicit TLS => STARTTLS)
      - "993:993"  # IMAP4 (implicit TLS)
    volumes:
      - ./mail/:/var/mail/
      - ./mail-state/:/var/mail-state/
      - ./mail-logs/:/var/log/mail/
      - ./config/:/tmp/docker-mailserver/
      - ./letsencrypt:/etc/letsencrypt
      - /etc/localtime:/etc/localtime:ro
      - /etc/ssl/private/deleted/:/tmp/dms/custom-certs/:ro
    restart: unless-stopped
    stop_grace_period: 1m
    # Uncomment if using `ENABLE_FAIL2BAN=1`:
    # cap_add:
    #   - NET_ADMIN
    healthcheck:
      test: "ss --listening --tcp | grep -P 'LISTEN.+:smtp' || exit 1"
      timeout: 3s
      retries: 0
docker exec -ti mailserver doveconf -f protocol=lmtp mail_plugins
mail_plugins =  quota sieve zlib
docker exec -ti mailserver doveconf -f protocol=imap mail_plugins
mail_plugins =  quota imap_quota zlib imap_zlib

Thank you again for your help. Everything seems to be working now and I am very happy!

As assumed user attempted to apply custom configuration and did so incorrectly. Closing again as not a bug.

User misconfiguration

What evidence do you have that this is a misconfiguration on my part? What in that dovecot.cf is misconfigured?

protocol imap {
	mail_plugins = $mail_plugins zlib imap_zlib
}

This was your problem. imap_zlib is only relevant to the IMAP protocol, while the zlib plugin should be available to the Dovecot protocol(s) for receiving mail (such as LMTP).

The zlib plugin docs do not scope to a specific protocol in their example. While the imap_zlib plugin docs do specifically use the imap protocol section for the example there. Easy mistake.


The configuration is included in dovecot config.

$ docker exec -it mailserver doveconf | grep zlib
  zlib_save = gz
  zlib_save_level = 9
  mail_plugins = " quota imap_quota zlib imap_zlib"

That did not properly check the config output for the mail_plugins setting:

# Filter config output by setting and include the line before and after the match(es):
$ doveconf | grep -B1 -A1 mail_plugins
mail_plugin_dir = /usr/lib/dovecot/modules
mail_plugins = " quota"
mail_prefetch_count = 0
--
protocol lmtp {
  mail_plugins = " quota sieve"
}
protocol lda {
  mail_plugins = " quota sieve"
}
protocol imap {
  mail_plugins = " quota imap_quota zlib imap_zlib"
}

# Just querying the setting defaults to the global setting, not protocol scoped:
$ doveconf mail_plugins
mail_plugins =  quota
# The setting for the LMTP protocol:
$ doveconf -f protocol=lmtp
mail_plugins =  quota sieve

Reproduction (with working compression)

Take this compose.yaml and run the shell commands that follow. No other external files are required, this all works in a self-contained compose.yaml example:

services:
  dms:
    image: mailserver/docker-mailserver:14.0.0
    hostname: mail.example.test
    # NOTE: Only used for the example for using swaks internally to send mail on port 25
    environment:
      PERMIT_DOCKER: container
    # NOTE: Normally you'd use `volumes` with external files instead of this Compose feature:
    configs:
      - source: dms-accounts
        target: /tmp/docker-mailserver/postfix-accounts.cf
      - source: dovecot-compression
        target: /tmp/docker-mailserver/dovecot.cf

# Using the Docker Compose `configs.content` feature instead of volume mounting separate files.
# NOTE: This feature requires Docker Compose v2.23.1 (Nov 2023) or newer:
# https://github.com/compose-spec/compose-spec/pull/446
# NOTE: `$` literals in `content` are repeated to escape and prevent the `compose.yaml` variable interpolation feature.
configs:
  # DMS expects an account to be configured to run, this example provisions one in advance:
  # Login credentials:
  # user: "john.doe@example.test" password: "secret"
  dms-accounts:
    content: |
      john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

  # LMTP protocol for Postfix => Dovecot delivery (enable zlib plugin for on-disk compression)
  # IMAP protocol for MUA (enable imap_zlib plugin for compressed network transfer to MUA client)
  dovecot-compression:
    content: |
      protocol lmtp {
        mail_plugins = $$mail_plugins zlib
      }
      protocol imap {
        mail_plugins = $$mail_plugins imap_zlib
      }

      # On-disk compression:
      plugin {
        zlib_save = gz
        zlib_save_level = 9
      }

      # IMAP network compression:
      plugin {
        imap_compress_deflate_level = 6
      }
# Start DMS container and shell into it:
$ docker compose up --force-recreate -d
$ docker compose exec -it dms bash

# Send a mail to the mailbox, it'll be stored compressed:
$ swaks --helo not-relevant.test --server 0.0.0.0 --port 25 --from hello@not-relevant.test --to john.doe@example.test

# Verify compression - GZIP will look like gibberish instead of plain-text output
$ cat /var/mail/example.test/john.doe/new/*

# Verify Dovecot config is correct for `mail_plugins` per protocol:
$ doveconf -f protocol=lmtp mail_plugins
mail_plugins =  quota sieve zlib
$ doveconf -f protocol=imap mail_plugins
mail_plugins =  quota imap_quota imap_zlib

As this is not officially supported by DMS, I do not deem this a bug.

Please instead raise a feature request to get enough user support (via ๐Ÿ‘ reacts) and ideally show a working configuration (since if you configure for LDA it shouldnโ€™t make any actual difference vs LMTP AFAIK).


Dovecot Compression feature is not on-disk storage compression

Actuallyโ€ฆ looking at the IMAP Dovecot docs I linked in the previous comment. I think @alex9434 has misunderstood what the plugin does?

https://doc.dovecot.org/2.3/configuration_manual/imap_compress/

Dovecot supports the IMAP COMPRESS (RFC 4978) extension.

It allows an IMAP client to dynamically enable stream compression for an IMAP session.

The IMAP mail client needs to use the extension when connecting to Dovecot. IMAP is used for mail retrieval, the compression is on-demand for transfer between Dovecot and the MUA. This isnโ€™t about delivery from Postfix with storage compression on-disk.

If you want compressed storage, an agnostic approach would be to use a filesystem that supports that. BTRFS and NTFS are common examples.


Closing - Advice for proceeding

I will close this given the above.

  • If you feel that I am mistaken, feel free to comment on why it should be re-opened.
  • If this feature was not misunderstood, open a feature request (you may be waiting a long time).
  • If you instead just want the functionality of this Dovecot plugin (as Iโ€™ve described it), open a new discussion on our Github Discussions page to seek community support for customizing DMS when our docs are not sufficient.

@georglauterbach and @polarathene I just received the notification for your last comment on github to my email and it is stored compressed (6,699 byte uncompressed and 2,572 bytes compressed)! So everything working! I understand that it can be frustrating for you to deal with all these misconfigurations. Its a complex subject with difficult documentation due to the multiple tools involved.

BTW: If at any point you want to include compression in your documentation, I am happy to write a few paragraphs based on the lessens learned.

Iโ€™m glad you figured this one out @polarathene, nice work!

Whatโ€™s really striking is the amount of user misconfigurations that we see in our issue tracker, really (no offense @alex9434, happens to everyone, and has happened to me too ๐Ÿ˜„).

No need for this to become heated ๐Ÿ˜ƒ We will have a look at your latest comments again re-evaluate. If it turns out that we misunderstood something, we will reconsider. Please do not open a feature request yet; I will re-open this first and we will decide here. On-Disk compression might be worthwhile, but Iโ€™d like for @polarathene to have a look at your feedback here because I donโ€™t know the details.