docker-mailserver: [NOT A BUG] Fresh 10.2.0 container crashes while booting: Failed to remove ClamAV configuration

Miscellaneous first checks

  • I checked that all ports are open and not blocked by my ISP / hosting provider.
  • I know that SSL errors are likely the result of a wrong setup on the user side and not caused by DMS itself. I’m confident my setup is correct.

Affected Component(s)

ClamAV

What happened and when does this occur?

When running a fresh container (no old mounted folders or files), the container crashes with the error:


[[  INF  ]]  Fixing Amavis permissions
[[ TASKS ]]  Cleaning up disabled ClamAV
rm: cannot remove '/etc/logrotate.d/clamav-*': No such file or directory
rm: cannot remove '/etc/cron.d/clamav-freshclam': No such file or directory
[  FATAL  ]  Failed to remove ClamAV configuration
[  ERROR  ]  Shutting down..


### What did you expect to happen?

```Markdown
The container should start normally

How do we replicate the issue?

Just run the container using the settings below.

DMS version

v10.2.0 (edge)

How much RAM is available to DMS explicitly?

less than 2GB

How many CPU cores are available?

less than 1 Core

Is DMS running in a virtualized environment?

… a virtual private server (VPS) (with virtual CPU cores)

What operating system is DMS running on?

Linux

What instruction set architecture is DMS running on?

x86_64 / AMD64

I/O - Persistent memory

What container orchestration tool are you using?

Docker Compose

Docker version

Docker version 20.10.8, build 3967b7d

Docker Compose version

docker-compose version 1.29.2, build 5becea4c

The output of uname -a

Linux srv02 5.4.0-84-generic #94-Ubuntu SMP Thu Aug 26 20:27:37 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Important environment variables

DMS_DEBUG=1

POSTMASTER_ADDRESS=postmaster@MYDOMAIN.com

SSL_TYPE=manual
SSL_CERT_PATH=/etc/ssl/pki/mail.MYDOMAIN.com.crt
SSL_KEY_PATH=/etc/ssl/pki/mail.MYDOMAIN.com.key

ENABLE_LDAP=1
LDAP_START_TLS=yes
LDAP_SERVER_HOST=dc1.internal.MYDOMAIN.com
LDAP_SEARCH_BASE=dc=internal,dc=MYDOMAIN,dc=com
LDAP_BIND_DN=cn=Administrator,cn=users,dc=internal,dc=MYDOMAIN,dc=com
LDAP_BIND_PW=Ch@ngeMe
# Specifies how ldap should be asked for users.
LDAP_QUERY_FILTER_USER=(&(objectclass=person)(mail=%s))
# Specifies how ldap should be asked for groups.
LDAP_QUERY_FILTER_GROUP=(|)
# Specifies how ldap should be asked for aliases.
LDAP_QUERY_FILTER_ALIAS=(|)
# Specifies how ldap should be asked for domains.
LDAP_QUERY_FILTER_DOMAIN=(mail=*@%s)

DOVECOT_TLS=yes
DOVECOT_USER_FILTER=(&(objectclass=person)(sAMAccountName=%n))
DOVECOT_USER_ATTRS==uid=%{ldap:uidNumber},=gid=6000,=home=/var/mail/%Ln,=mail=maildir:~/Maildir
DOVECOT_PASS_ATTRS=sAMAccountName=user,userPassword=password
DOVECOT_AUTH_BIND=yes

ENABLE_SASLAUTHD=1
SASLAUTHD_MECHANISMS=ldap
SASLAUTHD_LDAP_SERVER=dc1.internal.MYDOMAIN.com
SASLAUTHD_LDAP_BIND_DN=cn=Administrator,cn=users,dc=internal,dc=MYDOMAIN,dc=com
SASLAUTHD_LDAP_SEARCH_BASE=dc=internal,dc=MYDOMAIN,dc=com
SASLAUTHD_LDAP_FILTER=(&(sAMAccountName=%U)(objectClass=person))
SASLAUTHD_LDAP_START_TLS=yes
SASLAUTHD_LDAP_TLS_CHECK_PEER=yes
SASLAUTHD_LDAP_TLS_CACERT_FILE=/etc/ssl/self-signed-pki/pki/ca.crt

Relevant log output

[[  INF  ]]  Fixing Amavis permissions
[[ TASKS ]]  Cleaning up disabled ClamAV
rm: cannot remove '/etc/logrotate.d/clamav-*': No such file or directory
rm: cannot remove '/etc/cron.d/clamav-freshclam': No such file or directory
[  FATAL  ]  Failed to remove ClamAV configuration
[  ERROR  ]  Shutting down..

Other relevant information

**(do NOT miss number 2 below)**

1. Find the **Docker-Compose** file below:

version: '3.9'
services:
  mailserver:
    container_name: mailserver
    image: docker.io/mailserver/docker-mailserver:edge # TODO change the TAG to 10.2.0 when released.
    hostname: mail
    domainname: ${PUBLIC_DOMAIN_NAME}
    # dns: 1.1.1.1
    # To avoid conflicts with yaml base-60 float, DO NOT remove the quotation marks.
    # More information about the mailserver ports:
    # https://docker-mailserver.github.io/docker-mailserver/edge/config/security/understanding-the-ports/
    networks:
      - internal01
    ports:
      - "25:25"
      - "143:143"
      - "587:587"
      - "993:993"
    volumes:
      - mailserver-data:/var/mail/
      - mailserver-state:/var/mail-state/
      - mailserver-logs:/var/log/mail/
      - mailserver-config:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
      - publicly-trusted-pki:/etc/ssl/pki/:ro
    restart: always
    stop_grace_period: 1m
    cap_add: [ "NET_ADMIN", "SYS_PTRACE" ]
    env_file:
      - mailserver/mailserver.env
    environment:
      # The following variables will be passed from the .env (if exist) to override the variables in mailserver.env.
      - DMS_DEBUG
      - POSTMASTER_ADDRESS
      - SSL_TYPE
      - SSL_CERT_PATH
      - SSL_KEY_PATH
      - ENABLE_LDAP
      - LDAP_START_TLS
      - LDAP_SERVER_HOST
      - LDAP_SEARCH_BASE
      - LDAP_BIND_DN
      - LDAP_BIND_PW
      - LDAP_QUERY_FILTER_USER
      - LDAP_QUERY_FILTER_GROUP
      - LDAP_QUERY_FILTER_ALIAS
      - LDAP_QUERY_FILTER_DOMAIN
      - DOVECOT_TLS
      - DOVECOT_USER_FILTER
      - DOVECOT_USER_ATTRS
      - DOVECOT_PASS_ATTRS
      - DOVECOT_AUTH_BIND
      - ENABLE_SASLAUTHD
      - SASLAUTHD_MECHANISMS
      - SASLAUTHD_LDAP_SERVER
      - SASLAUTHD_LDAP_BIND_DN
      - SASLAUTHD_LDAP_SEARCH_BASE
      - SASLAUTHD_LDAP_FILTER
      - SASLAUTHD_LDAP_START_TLS
      - SASLAUTHD_LDAP_TLS_CHECK_PEER
      - SASLAUTHD_LDAP_TLS_CACERT_FILE


volumes:
  mailserver-data:
  mailserver-state:
  mailserver-logs:
  mailserver-config:

internal01 network and the volume of the public key are defined in another docker-compose.yml file. But this should not be an issue.

2. Make sure the container does not validate the CA in the ldap.conf file, as is seems that there is an issue here (the certificate seems to be refused even if it is valid). In simple words, execute the following on the host and this will do the job:

    docker exec mailserver sh -c 'echo "TLS_REQCERT   never" > /etc/ldap/ldap.conf'

What level of experience do you have with Docker and mail servers?

Trust me, I’m a (computer) engineer! [expert]

Code of conduct

Improvements to this form?

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 44 (44 by maintainers)

Most upvoted comments

always good if your container can just restart and recover itself.

Fair point. We should improve this. I agree that the error handling in this case is a but harsh.

@casperklein shall we emit a warning now, instead of logging a fatality?

I think Casper will address it as he worked on the previous PR that introduced this change. If not I’ll take care of it within the next day or so.

There doesn’t appear to be any reason to aggressively shutdown the container for those checks, the value is still there if using warning logs instead.

Thank you for raising the issue, it was helpful and it should be taken care of before the official release 😃

I’m not sure how you’re starting your containers, but here is the two ways to do this and avoid this problem:

  • docker run --rm
  • docker-compose up --force-recreate

And then add any extra options you normally have. The docker example will remove when container is stopped, while docker-compose will always recreate the containers.


to ignore such not-really-errors?

And if you later have something that would be a silent error because you didn’t create a fresh container?

I think it’s fair though. If other maintainers agree that it would be better to emit a warning instead. I don’t think we have a need to explicitly shut the container down for this error handling.

For me this behavior is a bit weird and uncommon that I need to recreate the whole container instead of restarting it. Can’t it be fixed, this is much nicer to be able to just restart the container?

You’d need a compelling case for it I think. As I explained in my last comment, it can be easy to accidentally get different behaviour that we can’t workaround, so it’s better to detect that early on.

If you really want to avoid this, you can override the error handling here, just remove the content on each line from || onwards. You can either build custom Dockerfile, or volume mount over it in the container.

Oooh ok now I understand, but why is this happning?

Your previous workflow (docker start/stop) might work, but it’s not best practice. With the new version, we introduced a stricter error checking. So these files couldn’t be deleted in the old version as well, but no error was shown.

Think of containers like the were toilet papers (should only be used once 😄). As a thumb rule, containers shouldn’t be stopped and started again.

Okay. Sorry. I didn’t know that 😞 I thought you would mistake .env file with mailserver.env

TIL that there is another crazy way to provide container variables 😄

So removing the container should be considered only if the container for a serious reason fails. Otherwise restart is the clean way.

Can you expand on the doc reference? That’s an area I look after, I’m going to take a guess you’re referring to restart: always in the config example?

That’s not necessary for cert renewals AFAIK, and it’s for the non-docker-mailserver containers, which I believe use the Docker API to restart the containers if necessary regardless of the setting.

When you restart a container, ENV changes don’t carry over IIRC, which can be unexpected. The clean way is re-creating a container as we advise as no state is persisted. Should the container encounter a failure and be eligible to restart itself, ideally it should be able to recover that way though, I’m not against that.

I am not sure if I understand you correctly, so please correct me if I did not get your point correctly.

Restarting the container (or at least the service that uses the certificate) is necessary in most cases (I am not sure what about postfix), but in general you would restart the services after renewing the certificates. That is what all other services (and docker containers) do. Actually I know no docker container that requires removing and recreating the container as a better solution than restarting. Of course if the container is stateless is good. I am just trying to say that nor restarting nor recreating should be better the other, they both should be good, imo.

@polarathene sorry I did not mean ignore literally. I just meant to log them as warnings without crashing. This is gonna be the best.

If it is possible please try to get it in this version, because this is directly related to a change done it it.

Thank you all for this discussion!

@polarathene @casperklein thank you for the explanation! At least I understand the whole thing now.

To be honest, I see it as a bad behavior, you don’t know what can happen with your server or container, and it is always good if your container can just restart and recover itself. I know it is good that you know exactly that something went wrong, but you may have that as a very emphasized log entry, but to just crash the container for such minor error is not something I would see, imho.

Can’t we have the ability to configure this behavior as environment variable to ignore such not-really-errors (I mean to log them as warnings)?

So something like this:

AUTO_RECOVER=1

or

STRICT_ERROR_HANDLING=0

@casperklein, recycling everything is always good 😄

This can happen for example, if you use docker-compose up and then hitting CTRL-C. This will just shutdown but not delete/destroy the container.

@MohammedNoureldin

The error occurs, when you do not fully shutdown (destroy) the container. On next start, the files are already gone because they were deleted previously.

In any case, you must run docker-compose down.

In doubt, check if the container is really gone with docker ps -a.

If you see something like this:

CONTAINER ID   IMAGE                               COMMAND                  CREATED         STATUS                     PORTS     NAMES
aad9522a93e9   mailserver/docker-mailserver:edge   "/usr/bin/dumb-init …"   7 minutes ago   Exited (0) 2 minutes ago             mailserver

then issue docker-compose down. Otherwise, THIS “already used” container will be used by docker-compose up again.

Stop. Variables in the file .env are not passed to the container. However, the vars in mailserver.env are passed to the container.

# .env
IT_WORKS=42
# docker-compose.yml
version: '3'

services:
  try_it:
    image: alpine
    container_name: try_it
    environment:
      - IT_WORKS
    tty: true

And now it works:

$ docker-compose up -d && docker exec -it try_it ash -c 'echo "${IT_WORKS}"'
42

You can add env_file for a custom env file, but you’ll still find vars passed through environment override it.

Good news: I can replicate your issue. Bad news: You probably didn’t do as adviced 😉