docker-mailserver: [BUG] Let's encrypt certificate renew not recognised. mailserver uses old one and not updated one.

Miscellaneous first checks

  • I checked that all ports are open and not blocked by my ISP / hosting provider.
  • I know that SSL errors are likely the result of a wrong setup on the user side and not caused by DMS itself. I’m confident my setup is correct.

Affected Component(s)

mailserver does not use renewed let’s encrypt certificate

What happened and when does this occur?

I'm using traefik v2 to get and renew certificates. The traefik json which holds the certificates has been mounted as volume into the mailserver docker container. When the certificate gets renewed by traefik (~4 weeks before expiry), the mailserver does not recognise this certificate and keeps using the old one. 

I checked certificate with https://www.checktls.com/TestReceiver and it still shows the old one. Opening the Website that uses the same hostname and certificate, it shows the renewed one. 

When I do a restart of the docker container, the new certificate is recognized and from there on used.

What did you expect to happen?

docker mailserver should recognise the updated / renewed certificate (by traefik2) without restarting the container.

How do we replicate the issue?

1. Setup traefik2, nginx webserver and docker mailserver as described in the docu
2. wait until let's encrypt certificate get's renewed by traefik2
3. check that mailserver does not pick renewed certificate
...

DMS version

v10.1.2

How much RAM is available to DMS explicitly?

more than 4GB

How many CPU cores are available?

less than 4 Cores

Is DMS running in a virtualized environment?

… a virtual private server (VPS) (with virtual CPU cores)

What operating system is DMS running on?

Linux

What instruction set architecture is DMS running on?

x86_64 / AMD64

I/O - Persistent memory

docker volume mount ext4

What container orchestration tool are you using?

Docker Compose

Docker version

No response

Docker Compose version

docker-compose version 1.28.5

The output of uname -a

Linux 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64 GNU/Linux

Important environment variables

Mount point within my docker-compose file: 

      - /data/acme/:/etc/letsencrypt/

I tried 

      - /data/acme/acme.json:/etc/letsencrypt/acme.json:ro

last time but same result.

Relevant log output

No response

Other relevant information

No response

What level of experience do you have with Docker and mail servers?

Trust me, I’m a (computer) engineer! [expert]

Code of conduct

Improvements to this form?

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 5
  • Comments: 43 (27 by maintainers)

Most upvoted comments

I updated to 10.3.0. Next auto-renew is around 4th of December. Therefore perfect timing from your end to release this! 😃

Will give you an update afterwards.

Renewal was successful! 😃 No manual restar required this time. Thanks a lot!!

Not Valid Before: Dec  5 22:42:21 2021 GMT
Not Valid After: Mar  5 22:42:20 2022 GMT

root@mail:/# ls -al /etc/letsencrypt/acme.json
-rw------- 1 root root 202986 Dec  7 00:42 /etc/letsencrypt/acme.json

root@mail:/# ls -al /etc/letsencrypt/live/
total 20
drwxr-xr-x 2 root root 4096 Nov 29 09:30 .
drwxr-xr-x 3 root root 4096 Nov 29 09:30 ..
-rw-r--r-- 1 root root 5946 Dec  7 00:42 fullchain.pem
-rw-r--r-- 1 root root 3243 Dec  7 00:42 key.pem

I’m 99% certain you’re experiencing the bug I mentioned. It looks like it may have expanded the certs into mail instead. Why it works after a reboot of the container though is a different problem. If you moved the contents of mail into mail.somedomain.com it would probably work after manually restarting postfix/dovecot. Do you have the ability to test that?

I copied the files from mail to mail.somedomain.com and restarted postfix/dovecot via supervisorctl restart dovecot / postfix. The renewed certificate is in use now:

Not Valid Before: Oct  6 07:20:34 2021 GMT
Not Valid After: Jan  4 07:20:33 2022 GMT

@zaphoodb how did it go? Renewal went smoothly this time? 😃

I’m very familiar with docker and running traefik as a reverse proxy (and using it to get letsencrypt certificates). I am using docker-mailserver since beginning of this year. So far everything worked. But I newer saw that docker-mailserver was picking up the changes to the acme.json (made by traefik) automatically. Only on a restart or re-create of the docker container, the updated certificate was used. Therefore I think it’s an option that this never worked as designed.

docker-mailserver docu about traefik: The file is also monitored for changes and will trigger a reload of the mail services (Postfix and Dovecot). This is newer saw happening.

That’s expected. The variable becomes whatever you set in the docker config’s ‘hostname’

How? We set it by whatever the returned value for a match in /etc/hosts is currently:

https://github.com/docker-mailserver/docker-mailserver/blob/34ba3c2a5b0f709a1680a779fda3da338535a063/target/scripts/helper-functions.sh#L302

So I’m not sure how $HOSTNAME would be mail when creating a folder with that value unless something was wrong in /etc/hosts config.


Output of docker exec mailserver bash -c 'cat /etc/hosts':

If there is no other lines with mail I’m not sure what is going on there 🤔

Maybe I misunderstood how $HOSTNAME has become mail and @NorseGaud is correct, but I’m not sure where that mishap occurs.

Ah… $HOSTNAME is only altered in the scripts scope, so echo $HOSTNAME in the container would be misleading (since this value is output by hostname default command that matches /proc/sys/kernel/hostname which was set for the container).

Yep,we don’t control HOSTNAME, it’s a docker thing. We override it in the scripts though. Sorry for brevity; traveling

That’s expected. The variable becomes whatever you set in the docker config’s ‘hostname’

How? We set it by whatever the returned value for a match in /etc/hosts is currently:

https://github.com/docker-mailserver/docker-mailserver/blob/34ba3c2a5b0f709a1680a779fda3da338535a063/target/scripts/helper-functions.sh#L302

So I’m not sure how $HOSTNAME would be mail when creating a folder with that value unless something was wrong in /etc/hosts config.


Output of docker exec mailserver bash -c 'cat /etc/hosts':

If there is no other lines with mail I’m not sure what is going on there 🤔

Maybe I misunderstood how $HOSTNAME has become mail and @NorseGaud is correct, but I’m not sure where that mishap occurs.

Ah… $HOSTNAME is only altered in the scripts scope, so echo $HOSTNAME in the container would be misleading (since this value is output by hostname default command that matches /proc/sys/kernel/hostname which was set for the container).

I double checked and in my docker-compose.yml I have set following:

    hostname: mail
    domainname: somedomain.com

Output of docker exec mailserver bash -c 'cat /etc/hosts':

<snip>
172.30.0.2      mail.somedomain.com mail

I do not use any loadbalancer (e.g. traefik) in front of the mailserver container. Therefore docker creates an own network just for this container:

<snip>
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "f3cec0e45aafb6db4515d6961f345664d3710550546f839301e1b198f77dfa0d",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "110/tcp": null,
                "143/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "143"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "143"
                    }
                ],
                "25/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "25"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "25"
                    }
                ],
                "4190/tcp": null,
                "465/tcp": null,
                "587/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "587"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "587"
                    }
                ],
                "993/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "993"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "993"
                    }
                ],
                "995/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/f3cec0e45aaf",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "docker-mailserver_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "mailserver",
                        "4b183787b840",
                        "mail"
                    ],
                    "NetworkID": "3df99af086451b086146e71290227ff2dd981a80211e2efd30c1be30ede7caf2",
                    "EndpointID": "b66dc22d9c50a5ef00a5975fad797d4e2ea0ad2d6d48fbb5ad84500d9b06fe90",
                    "Gateway": "172.30.0.1",
                    "IPAddress": "172.30.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:1e:00:02",
                    "DriverOpts": null
                }
            }
        }
    }

Executing this docker exec mailserver bash -c 'openssl x509 -noout -text -in /etc/letsencrypt/live/"${HOSTNAME}"/fullchain.pem raises an error now: Can't open /etc/letsencrypt/live/mail/fullchain.pem for reading, No such file or directory. That’s because I updated the docker container accidentially. This removed the mail folder which the update certificates process created (and did not overwrite mail.somedomain.com) instead. But folder mail.somedomain.com exists and the correct certificate is used from there.

hostname: mail domainname: somedomain.com

Other (relevant) environment variables: environment: - SSL_TYPE=letsencrypt - SSL_DOMAIN=mail.somedomain.com

OVERRIDE_HOSTNAME is not set in my docker-compose.yml

@zaphoodb

In your setup, you should not need to use SSL_DOMAIN. Not that it should make any difference for you.

Executing docker exec mailserver bash -c 'openssl x509 -noout -text -in /etc/letsencrypt/live/"${HOSTNAME}"/fullchain.pem' (HOSTNAME is “mail”)

Here, you are saying HOSTNAME the environment variable in the container (docker exec mailserver bash -c 'echo $HOSTNAME') is mail? Not just your hostname: setting in docker-compose.yml?

I don’t think that should happen, and I think that our PR fixes won’t resolve that issue.

That’s expected. The variable becomes whatever you set in the docker config’s ‘hostname’