docker-mailserver: [BUG] rspamd forgets it learned spam and ham when docker-compose down and up is issued

Miscellaneous first checks

I checked that all ports are open and not blocked by my ISP / hosting provider.
I know that SSL errors are likely the result of a wrong setup on the user side and not caused by DMS itself. I’m confident my setup is correct.

Affected Component(s)

rspamd will not easily learn to recognize spam

What happened and when does this occur?

I have trained rspamd with two email-folders containing spam or ham like so

First get stats on learned spam `docker exec -it mailserver sh rspamc stat

Statfile: BAYES_SPAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 1; users: 1; languages: 0 Statfile: BAYES_HAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 6; users: 1; languages: 0 Total learns: 7 `

Then I train rspamd with a folder full of spam, and a folder full of spam

rspamc learn_spam /var/mail/fanscees.com/hanscees/.spam/cur rspamc stat Statfile: BAYES_SPAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 413; users: 1; languages: 0 Statfile: BAYES_HAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 6; users: 1; languages: 0 Total learns: 419 Same with ham training rspamc learn_ham /var/mail/hanscees.com/hanscees/.Ham/cur rspamc stat Statfile: BAYES_SPAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 413; users: 1; languages: 0 Statfile: BAYES_HAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 354; users: 1; languages: 0 Total learns: 767

I also have enabled that rspamd learns from users when they move email in and out the inbox.junk folder: see attached txt script rspamd-patches.txt

Problem is now, after doing a docker-compose down, all learned material seems gone.

docker-compose down docker-compose up -d Statfile: BAYES_SPAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 0; users: 0; languages: 0 Statfile: BAYES_HAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 0; users: 0; languages: 0 Total learns: 0

I think something should be added to the mailstate that is preserved, but no idea what.

What did you expect to happen?

I expect learned spam to persist over a reset of the docker image

How do we replicate the issue?

do what I did above

DMS version

1301

What operating system is DMS running on?

Linux

Which operating system version?

photon 4

What instruction set architecture is DMS running on?

AMD64 / x86_64

What container orchestration tool are you using?

Docker

docker-compose.yml

services:
  mailserver:
    image: docker.io/mailserver/docker-mailserver:edge
    #image: docker.io/mailserver/docker-mailserver:latest
    container_name: mailserver
    # If the FQDN for your mail-server is only two labels (eg: example.com),
    # you can assign this entirely to `hostname` and remove `domainname`.
    hostname: mail
    domainname: fanscees.net
    env_file: mailserver.env
    # More information about the mail-server ports:
    # https://docker-mailserver.github.io/docker-mailserver/edge/config/security/understanding-the-ports/
    # To avoid conflicts with yaml base-60 float, DO NOT remove the quotation marks.
    ports:
      - "25:25"    # SMTP  (explicit TLS => STARTTLS)
      - "143:143"  # IMAP4 (explicit TLS => STARTTLS)
      - "465:465"  # ESMTP (implicit TLS)
      - "587:587"  # ESMTP (explicit TLS => STARTTLS)
      - "993:993"  # IMAP4 (implicit TLS)
      - "995:995"  # pop3 (implicit TLS)
      - "110:110"  # pop3 (implicit TLS)
    volumes:
      #- etc:/etc
      - maildata:/var/mail/
      - mailstate:/var/mail-state/
      - log:/var/log/
      - Tconfig:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
      - ssl:/tmp/ssl
    restart: always
    stop_grace_period: 1m
    environment:
     # If you need SSL connection, you can provide your own certificates
     - SSL_TYPE=manual
     - SSL_CERT_PATH=/tmp/ssl/fullchain.pem
     - SSL_KEY_PATH=/tmp/ssl/privkey.pem
    cap_add:
      - NET_ADMIN
    healthcheck:
      test: "ss --listening --tcp | grep -P 'LISTEN.+:smtp' || exit 1"
      timeout: 3s
      retries: 0
volumes:
  maildata:
  mailstate:
  Tconfig:
  log:
  ssl:

Relevant log output

No response

Other relevant information

I have also enabled the auto-learn rule on spam form your documentation

What level of experience do you have with Docker and mail servers?

I am inexperienced with docker
I am rather experienced with docker
I am inexperienced with mail servers
I am rather experienced with mail servers
I am uncomfortable with the CLI
I am rather comfortable with the CLI

Code of conduct

I have read this project’s Code of Conduct and I agree
I have read the README and the documentation and I searched the issue tracker but could not find a solution

Improvements to this form?

No response

About this issue

Original URL
State: closed
Created a year ago
Comments: 15 (15 by maintainers)

Commits related to this issue

adjust Redis configuration We are starting Redis now properly by supplying a configuration file. This way, Redis will also properly dump its contents to `/var/mail-state/redis/` and persists its data... — committed to docker-mailserver/docker-mailserver by georglauterbach a year ago

Most upvoted comments

I noticed this as well, and I will look into it after my current PRs are merged.

georglauterbach on Mar 1, 2023

I couldnt help myself and tried to fix form the userpatch script. Alas it massively failed. To change how redis starts you need to change supervisorctl and that probably starts before the user script. So you have to reload it and then many things go wrong.

So I learned a lot about the upstart scripts 😃

The changes I tried are below, but they fail so dont try this at home. Perhaps when George later on picks this up they might help.

This is what I added to the userscript.

### so adjusting redis files should be like
#adjust /var/lib where persistent database will be
mkdir /var/mail-state/lib-redis  # only once
ln -s /var/mail-state/lib-redis /var/lib/redis

#change redis config file
sed -i  "s|bind 127.0.0.1 ::1|bind 127.0.0.1|" /etc/redis/redis.conf
sed -i  "s|daemonize yes|daemonize no|" /etc/redis/redis.conf
## it might also need a change in the config part about supervised

# adjust supervisor-app.conf redis command
sed -i  "s|--daemonize no --bind 127.0.0.1 --port 6379 --loglevel warning|/etc/redis/redis.conf --daemonize no|" /etc/supervisor/conf.d/supervisor-app.conf

# this fails dont do it
#supervisorctl reload
#supervisorctl stop redis
#supervisorctl start redis

also I dont know why redis and rspamd do not log to /var/log. Makes it harder to troubleshoot, but perhaps there is a gooid reason for it I dont know.

hanscees on Mar 3, 2023

This all makes me very happy 👍

hanscees on Mar 3, 2023

PR is the abbreviation for Pull Request 😃

georglauterbach on Mar 3, 2023

#3143 will resolve this issue. You can have a look at the changes I made there to get an idea what is required. You were already very close @hanscees, thanks for your insights!

georglauterbach on Mar 3, 2023

However these were already configured explicitly via the args in the supervisor config, which you then replace with a config path and the daemonize opt-out again for some reason?

The whole point is to get redis to save what it learns from rspamd learning spam and ham. To do that I used the config file, since you need a lot of options for that. The database does not get saved very often but that’s not a big deal, it does save by default when shutting down. It saves the database in mail-state with the changes and link command.

hanscees on Mar 3, 2023