portainer: failed opening store | error="invalid database" in docker swarm

Before you start please confirm the following.

Problem Description

Hello, I have a 3 node Docker Swarm (all are Master Nodes) and I am using cephfs mounts (mounted in /var/data) on all my swarm nodes for docker volume syncing between my nodes.

Mount command in /etc/fstab:

admin@.data=/    /var/data   ceph   mon_addr=192.168.1.100:6789/192.168.1.101:6789/192.168.0.102:6789,noatime,_netdev   0   0

I am using a slightly modified version of https://downloads.portainer.io/ee2-21/portainer-agent-stack.yml

My portainer.yaml file:

version: '3.2'

services:
  agent:
    image: portainer/agent:2.21.1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - agent_network
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]

  portainer:
    image: portainer/portainer-ee:2.21.1
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - "9443:9443"
      - "9000:9000"
      - "8000:8000"
    volumes:
      - /var/data/portainer:/data
    networks:
      - agent_network
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

networks:
  agent_network:
    driver: overlay
    attachable: true

If I am deploying the service (sudo docker stack deploy portainer -c portainer.yaml) for the first time everything is fine.

But if I shutdown the node where the portainer_portainer (Web-GUI container) is running the service is deploying the portainer_portainer container on one of the two remaining nodes in my docker swarm. But then I get the following error:

Command I used to log the events of the container: sudo docker service logs portainer_portainer -f

Output:

portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:98 > failed opening store | error="invalid database"

My guess is an incorrect permissions on the directories /var/data and /var/data/portainer

Here are the permissions for the folder /var/data and /var/data/portainer (755) owner and group of all file is root:

user@dockerswm1:~$ sudo ls -la /var/data
total 4
drwxr-xr-x  3 root root    1 Sep 13 23:46 .
drwxr-xr-x 14 root root 4096 Sep 13 23:40 ..
drwxr-xr-x  2 root root    0 Sep 13 23:46 portainer

file permissions inside /var/data/portainer:

user@dockerswm1:~$ sudo ls -la /var/data/portainer
total 65
drwxr-xr-x 8 root root     9 Sep 13 23:50 .
drwxr-xr-x 3 root root     1 Sep 13 23:46 ..
drwx------ 2 root root     0 Sep 13 23:50 bin
drwx------ 2 root root     2 Sep 13 23:50 certs
drwx------ 2 root root     1 Sep 13 23:50 chisel
drwx------ 2 root root     0 Sep 13 23:50 compose
drwx------ 2 root root     1 Sep 13 23:50 docker_config
-rw------- 1 root root 65536 Sep 13 23:50 portainer.db
-rw------- 1 root root   227 Sep 13 23:50 portainer.key
-rw------- 1 root root   190 Sep 13 23:50 portainer.pub
drwx------ 2 root root     0 Sep 13 23:50 tls

Expected Behavior

portainer_portainer container should find the portainer.db inside /var/data/portainer and start the container on another docker swarm node after one node in the docker swarm is shutdown

Actual Behavior

portainer_portainercontainer can not start on one of the two remaining nodes:

Command I used to log the events of the container: sudo docker service logs portainer_portainer -f

Output:

portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:98 > failed opening store | error="invalid database"

Steps to Reproduce

1. Build Docker Swarm

<span style="color:#60A917">dockerswm1 (192.168.0.100)</span> init Swarm

Initialize the Docker Swarm:

!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:

sudo docker swarm init --advertise-addr 192.168.0.100

<span style="color:#0050EF">dockerswm2 (192.168.0.101)</span> add to Swarm as Manager

Now the machine dockerswm2 is added to the swarm as an additional manager node:

Please add the machine as Master-Node to the docker swarm !!!

!!! Only on machine <span style="color:#0050EF">dockerswm2</span> !!!:

docker swarm join \
    --token SWMTKN-1-59egwe8qangbzbqb3ryawxzk3jn97ifahlsrw01yar60pmkr90-bdjfnkcflhooyafetgjod97sz \
    192.168.0.100:2377

<span style="color:#A20025">dockerswm3 (192.168.0.102)</span> add to Swarm as Manager

Now the machine dockerswm3 is added to the swarm as an additional manager node:

Please add the machine as Master-Node to the docker swarm !!!

!!! Only on machine <span style="color:#A20025">dockerswm3</span> !!!:

docker swarm join \
    --token SWMTKN-1-59egwe8qangbzbqb3ryawxzk3jn97ifahlsrw01yar60pmkr90-bdjfnkcflhooyafetgjod97sz \
    192.168.0.100:2377


2. Setup ceph storage cluster

First we need to install the cephadm tool:

The following commands are repeated for the following systems (<span style="color:#60A917">dockerswm1</span>; <span style="color:#0050EF">dockerswm2</span>; <span style="color:#A20025">dockerswm3</span>):

sudo apt install -y cephadm

Install the ceph cli tool via cephadm tool:

The following commands are repeated for the following systems (<span style="color:#60A917">dockerswm1</span>; <span style="color:#0050EF">dockerswm2</span>; <span style="color:#A20025">dockerswm3</span>):

sudo apt install -y ceph-common

With the following command the cephadm tool is used for bootstraping a ceph cluster:

!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:

sudo cephadm bootstrap --mon-ip 192.168.0.100

Now we add the other machines to the ceph cluster.

But before that we need to transfer three files from the master node (dockerswm1) to the two non-master nodes (dockerswm2 and dockerswm3):

Path on master Path on non-master
/etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.client.admin.keyring
/etc/ceph/ceph.conf /etc/ceph/ceph.conf
/etc/ceph/ceph.pub /root/.ssh/authorized_keys

Please replace root user with the username that can access the machine via ssh!!! Please replace REMOTE_SSH_USER_PASSWORD with the password of the user that can access the machine via ssh!!!

!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:

sudo cat /etc/ceph/ceph.pub | ssh root@192.168.0.101 "tee /tmp/authorized_keys && echo 'REMOTE_SSH_USER_PASSWORD' | sudo -S sh -c 'mkdir -p /root/.ssh && cat /tmp/authorized_keys | tee -a /root/.ssh/authorized_keys > /dev/null && rm /tmp/authorized_keys'"
sudo cat /etc/ceph/ceph.pub | ssh root@192.168.0.102 "tee /tmp/authorized_keys && echo 'REMOTE_SSH_USER_PASSWORD' | sudo -S sh -c 'mkdir -p /root/.ssh && cat /tmp/authorized_keys | tee -a /root/.ssh/authorized_keys > /dev/null && rm /tmp/authorized_keys'"

sudo ceph orch host add dockerswm2 192.168.0.101 --labels _admin
sudo ceph orch host add dockerswm3 192.168.0.102 --labels _admin

Now we add OSDs to our ceph cluster.

!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:

sudo ceph orch apply osd --all-available-devices

This will identify any unpartitioned, unmounted disks attached to each participating node, and configure these disks as OSDs.


Now we create a cephfs volume in our cluster by running the following command:

!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:

sudo ceph fs volume create data

You can watch the progress by running sudo ceph fs ls (to see the fs is configured), and sudo ceph -s to wait for HEALTH_OK.


At last we mount the CephFS volume under /var/data on ervery docker swarm node:

The following commands are repeated for the following systems (<span style="color:#60A917">dockerswm1</span>; <span style="color:#0050EF">dockerswm2</span>; <span style="color:#A20025">dockerswm3</span>):

sudo mkdir /var/data
sudo sh -c 'echo "
# Mount cephfs volume
admin@.data=/    /var/data   ceph   mon_addr=192.168.1.100:6789/192.168.1.101:6789/192.168.0.102:6789,noatime,_netdev   0   0" >> /etc/fstab'
sudo mount -a
sudo systemctl daemon-reload

You can now create new files in the directory /var/data, which will then be synchronized to all machines.



3. Install Portainer

Portainer is a container for managing Docker containers via a graphical web interface.

Create a folder for the Portainer container (persistend data) (container volume):

!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:

sudo mkdir /var/data/portainer

Now a YAML file (docker-compose) is created to start the Portainer container in the Swarm environment. Please use nano to create the file portainer.yaml file and fill it with the code contained in the following code block and save it.

!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:

nano portainer.yaml

Insert the following content into the nano editor, save the file and close the nano editor:

version: '3.2'

services:
  agent:
    image: portainer/agent:2.21.1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - agent_network
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]

  portainer:
    image: portainer/portainer-ee:2.21.1
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - "9443:9443"
      - "9000:9000"
      - "8000:8000"
    volumes:
      - /var/data/portainer:/data
    networks:
      - agent_network
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

networks:
  agent_network:
    driver: overlay
    attachable: true


4. Shutdown docker node where the portainer_portainer container is currently running

!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:

sudo shutdown now


5. Portainer stack is restarting the portainer_portainer on one of the two remaining docker nodes but this error is appearing:

Command I used to log the events of the container: sudo docker service logs portainer_portainer -f

Output:

portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:98 > failed opening store | error="invalid database"

Portainer logs or screenshots

Command I used to log the events of the container: sudo docker service logs portainer_portainer -f

Output:

portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.qv1leusuuuig@dockerswm2    | 2024/09/13 09:11PM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:98 > failed opening store | error="invalid database"

Portainer version

2.21.1

Portainer Edition

Business Edition (EE)

Platform and Version

Docker Swarm 27.2.1

OS and Architecture

Ubuntu 24.04

Browser

Firefox 130.0

What command did you use to deploy Portainer?

version: '3.2'

services:
  agent:
    image: portainer/agent:2.21.1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - agent_network
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]

  portainer:
    image: portainer/portainer-ee:2.21.1
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - "9443:9443"
      - "9000:9000"
      - "8000:8000"
    volumes:
      - /var/data/portainer:/data
    networks:
      - agent_network
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

networks:
  agent_network:
    driver: overlay
    attachable: true

Additional Information

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 8
  • Comments: 45

Most upvoted comments

Hmmm, i did restart my full 3 node swarm multiple times and i don’t have any issue with the restart.

I also think it’s related to locks, maybe it work just fine one mine cuz it saved and unlocked the DB before i shut it down.

If you want to test here is all the options i have on my nodes noasyncreaddir,rbytes,wsync,read_from_replica=balance,sync, tho i don’t think all of them do something.

And also i have edited one conf in the ceph cluster directly the fuse_disable_pagecache that i setup to true for the clients

Sorry for my late response. @ibaraki-douji, I have now implemented your tips in my cluster:

(I have setup my cluter from the ground up)

  1. Repeated all the steps below “1. Build Docker Swarm” from my initial bug report.

  2. Put the following content in the config file (repeated on each node):

    {
      "storage-driver": "fuse-overlayfs"
    }
    
  3. sudo apt install fuse-overlayfs (repeated on each node)

  4. reboot each node.

  5. Verified on each node that the fuse-overlayfs drivers are used:

    sudo docker info --format '{{.Driver}}'
    

    Output on is node is: fuse-overlayfs

  6. Repeated all the steps below “2. Setup ceph storage cluster” from my initial bug report.

  7. Added your suggested mount options in the /etc/fstab (repeated on each node):

    sudo mkdir /var/data
    sudo sh -c 'echo "
    # Mount cephfs volume
    admin@.data=/    /var/data   ceph   mon_addr=192.168.1.100:6789/192.168.1.101:6789/192.168.0.102:6789,noatime,_netdev,noasyncreaddir,rbytes,wsync,read_from_replica=balance,sync   0   0" >> /etc/fstab'
    sudo mount -a
    sudo systemctl daemon-reload
    
  8. Set ceph parameter fuse_disable_pagecache to true (ONLY ON ONE NODE):

sudo ceph config set client fuse_disable_pagecache true

This commmand need a client name (in my example it is called client). Is this client name correct ??? My three node have the following names (dockerswm1, dockerswm2 and dockerswm3)

  1. Check if parameter is set (repeated on each node):

    sudo ceph config dump | grep client
    

    Output on is node is: client advanced fuse_disable_pagecache true

  2. Repeated all the steps below “3. Install Portainer” from my initial bug report.

Unfortunately I get the same error with the portainer database after I shutdown the node where the docker service portainer_portainer was running:

`sudo docker service logs portainer_portainer -f

portainer_portainer.1.74733w2je0ce@dockerswm2    | 2024/10/13 07:20PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:332 > encryption key file not present | filename=portainer
portainer_portainer.1.74733w2je0ce@dockerswm2    | 2024/10/13 07:20PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:356 > proceeding without encryption key |
portainer_portainer.1.74733w2je0ce@dockerswm2    | 2024/10/13 07:20PM INF github.com/portainer/portainer/api/database/boltdb/db.go:124 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.74733w2je0ce@dockerswm2    | 2024/10/13 07:20PM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:102 > failed opening store | error="invalid database"

i have basically the exact same setup but with glusterfs as shared-fs backing the swarm, there the portainer-stuff works just fine. i wanted to migrate to ceph because a thousand other things dont work well.

I was doing the same thing until Gluster just didnt cut it anymore and i moved to Ceph. for what its worth, while this potential bug is evaluated, I moved Portainer to a raspberry pi I use for critical base services like DNS and just connected it using the Agent instead. I’d imagine you could easily run a copy of portainer on any old thing or even your primary workstation and tie it back using the agent temporarily. then once this is resolved we can all migrate our configs and portainer volumes back to the swarm.

I find it odd that this database in particular is having this issue on Ceph given that my postgres and sqlite db’s are having no issues.

I have portainer running in an Ubuntu 24.04 Virtual Machine. Ceph version is 19.2.0.

I have switched my portainer data path to a local folder within the VM and … SURPRISE … the issue is the same! It seems that this doesn’t even have anything to do with ceph, a very simple deploy using this:

sudo docker run -d --name portainer --restart=always -p 8000:8000 -p 9000:9000 -p 9443:9443 -v /var/run/docker.sock:/var/run/docker.sock -v /home/main/portainer:/data portainer/portainer-ce:latest

results in a setup that will compromise its own database. What is going on here?

same issue here; other db applications on cephfs seem to be unaffected e.g. slite3 or mysql. pve-cluster with 3 nodes and ceph. docker 27.3.1, portainer CE 2.21.4, ubuntu 24.04, ceph 18.2.4 will give these “mitigations” a shot.

my compose also looks a bit different:

portainer:
    image: portainer/portainer-ce:latest
    [..]
    volumes:
      - type: bind
        source: /mnt/cephfs/dswarm/portainer
        target: /data
    [..]

edit: the flags etc. did not help.

i noticed i can provoke the crippled database by just creating a backup via gui.

portainer_portainer.1.68ernx7z1kod@docker-node-3    | 2024/10/28 12:00PM INF github.com/portainer/portainer/api/datastore/backup.go:25 > Backing up database | from=/data/portainer.db to=/data/backup/2024-10-28_12-00-50/portainer.db
portainer_portainer.1.68ernx7z1kod@docker-node-3    | 2024/10/28 12:00PM INF github.com/portainer/portainer/api/database/boltdb/db.go:147 > closing PortainerDB |
portainer_portainer.1.68ernx7z1kod@docker-node-3    | 2024/10/28 12:00PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.68ernx7z1kod@docker-node-3    | 2024/10/28 12:01PM ERR github.com/portainer/portainer/api/internal/snapshot/snapshot.go:243 > background schedule error (environment snapshot) | error="database not open"

after updating the service:

portainer_portainer.1.dawyq1e4vpc4@docker-node-3    | 2024/10/28 12:07PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
portainer_portainer.1.dawyq1e4vpc4@docker-node-3    | 2024/10/28 12:07PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
portainer_portainer.1.dawyq1e4vpc4@docker-node-3    | 2024/10/28 12:07PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.dawyq1e4vpc4@docker-node-3    | 2024/10/28 12:07PM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:98 > failed opening store | error="invalid database"

i have basically the exact same setup but with glusterfs as shared-fs backing the swarm, there the portainer-stuff works just fine. i wanted to migrate to ceph because a thousand other things dont work well.

unfortunately spoke to soon. after a reboot of the cluster it’s failing again. it seems to be related to file locking but im not certain yet.

My solution was to move away from cephfs for storing the database and instead have the container exclusively mount an RBD volume formatted to ext4. Running super stable throughout updates, upgrades, reboots, and host switching for months now. So its definitely some kind of compatibility issue with cephfs - which isn’t surprising. Clustered filesystems are complicated and not really intended for complicated workloads like databases.

I’ve got the same problem (Ubuntu 24.04 and Ceph 19.2 on Proxmox). Tried several inputs from this thread but nothing worked. My current solution is to use a CIFS volume mapped to a high available samba server (the storage is needed for the windows container hosts).

Hello @SuitDeer ,

Thank you for your detailed post. I’m working with the same setup as many here:

  • 1 manager
  • 2 workers

Shared files via CephFS but I got the same issue as you described…

Did you find a solution / workaround?

It’s my first time setting up a swarm cluster. For the moment it’s in a LAB / TEST environnement but I very feel unconfident about using Portainer in production… Even more after seeing that you’ve reported this issue back in September 2024 and we are in January 2025 and it’s not resolved yet…

Thank you!

Today I have build my docker swarm cluster from the ground up again. Now I am using the following tutorial: https://www.youtube.com/watch?v=nmas_zbcMeU

But with microceph (ubuntu) I get the same error 🤔

adding sync to the fstab line seems to have worked. I was able to shutdown and restart the container without encountering the db issue in the logs

hmm, strange, do you have removed the old invalid database after the fuse change ?

also are you using some options for your mount ?? if not try to add some options to remove all the async R/W https://docs.ceph.com/en/reef/man/8/mount.ceph/