portainer: failed opening store | error="invalid database" in docker swarm
Before you start please confirm the following.
- Yes, I’ve searched similar issues on GitHub.
- Yes, I’ve checked whether this issue is covered in the Portainer documentation or knowledge base.
Problem Description
Hello,
I have a 3 node Docker Swarm (all are Master Nodes) and I am using cephfs mounts (mounted in /var/data) on all my swarm nodes for docker volume syncing between my nodes.
Mount command in /etc/fstab:
admin@.data=/ /var/data ceph mon_addr=192.168.1.100:6789/192.168.1.101:6789/192.168.0.102:6789,noatime,_netdev 0 0
I am using a slightly modified version of https://downloads.portainer.io/ee2-21/portainer-agent-stack.yml
My portainer.yaml file:
version: '3.2'
services:
agent:
image: portainer/agent:2.21.1
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/volumes:/var/lib/docker/volumes
networks:
- agent_network
deploy:
mode: global
placement:
constraints: [node.platform.os == linux]
portainer:
image: portainer/portainer-ee:2.21.1
command: -H tcp://tasks.agent:9001 --tlsskipverify
ports:
- "9443:9443"
- "9000:9000"
- "8000:8000"
volumes:
- /var/data/portainer:/data
networks:
- agent_network
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
networks:
agent_network:
driver: overlay
attachable: true
If I am deploying the service (sudo docker stack deploy portainer -c portainer.yaml) for the first time everything is fine.
But if I shutdown the node where the portainer_portainer (Web-GUI container) is running the service is deploying the portainer_portainer container on one of the two remaining nodes in my docker swarm. But then I get the following error:
Command I used to log the events of the container:
sudo docker service logs portainer_portainer -f
Output:
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:98 > failed opening store | error="invalid database"
My guess is an incorrect permissions on the directories /var/data and /var/data/portainer
Here are the permissions for the folder /var/data and /var/data/portainer (755) owner and group of all file is root:
user@dockerswm1:~$ sudo ls -la /var/data
total 4
drwxr-xr-x 3 root root 1 Sep 13 23:46 .
drwxr-xr-x 14 root root 4096 Sep 13 23:40 ..
drwxr-xr-x 2 root root 0 Sep 13 23:46 portainer
file permissions inside /var/data/portainer:
user@dockerswm1:~$ sudo ls -la /var/data/portainer
total 65
drwxr-xr-x 8 root root 9 Sep 13 23:50 .
drwxr-xr-x 3 root root 1 Sep 13 23:46 ..
drwx------ 2 root root 0 Sep 13 23:50 bin
drwx------ 2 root root 2 Sep 13 23:50 certs
drwx------ 2 root root 1 Sep 13 23:50 chisel
drwx------ 2 root root 0 Sep 13 23:50 compose
drwx------ 2 root root 1 Sep 13 23:50 docker_config
-rw------- 1 root root 65536 Sep 13 23:50 portainer.db
-rw------- 1 root root 227 Sep 13 23:50 portainer.key
-rw------- 1 root root 190 Sep 13 23:50 portainer.pub
drwx------ 2 root root 0 Sep 13 23:50 tls
Expected Behavior
portainer_portainer container should find the portainer.db inside /var/data/portainer and start the container on another docker swarm node after one node in the docker swarm is shutdown
Actual Behavior
portainer_portainercontainer can not start on one of the two remaining nodes:
Command I used to log the events of the container:
sudo docker service logs portainer_portainer -f
Output:
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:98 > failed opening store | error="invalid database"
Steps to Reproduce
1. Build Docker Swarm
<span style="color:#60A917">dockerswm1 (192.168.0.100)</span> init Swarm
Initialize the Docker Swarm:
!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:
sudo docker swarm init --advertise-addr 192.168.0.100
<span style="color:#0050EF">dockerswm2 (192.168.0.101)</span> add to Swarm as Manager
Now the machine dockerswm2 is added to the swarm as an additional manager node:
Please add the machine as Master-Node to the docker swarm !!!
!!! Only on machine <span style="color:#0050EF">dockerswm2</span> !!!:
docker swarm join \
--token SWMTKN-1-59egwe8qangbzbqb3ryawxzk3jn97ifahlsrw01yar60pmkr90-bdjfnkcflhooyafetgjod97sz \
192.168.0.100:2377
<span style="color:#A20025">dockerswm3 (192.168.0.102)</span> add to Swarm as Manager
Now the machine dockerswm3 is added to the swarm as an additional manager node:
Please add the machine as Master-Node to the docker swarm !!!
!!! Only on machine <span style="color:#A20025">dockerswm3</span> !!!:
docker swarm join \
--token SWMTKN-1-59egwe8qangbzbqb3ryawxzk3jn97ifahlsrw01yar60pmkr90-bdjfnkcflhooyafetgjod97sz \
192.168.0.100:2377
2. Setup ceph storage cluster
First we need to install the cephadm tool:
The following commands are repeated for the following systems (<span style="color:#60A917">dockerswm1</span>; <span style="color:#0050EF">dockerswm2</span>; <span style="color:#A20025">dockerswm3</span>):
sudo apt install -y cephadm
Install the ceph cli tool via cephadm tool:
The following commands are repeated for the following systems (<span style="color:#60A917">dockerswm1</span>; <span style="color:#0050EF">dockerswm2</span>; <span style="color:#A20025">dockerswm3</span>):
sudo apt install -y ceph-common
With the following command the cephadm tool is used for bootstraping a ceph cluster:
!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:
sudo cephadm bootstrap --mon-ip 192.168.0.100
Now we add the other machines to the ceph cluster.
But before that we need to transfer three files from the master node (dockerswm1) to the two non-master nodes (dockerswm2 and dockerswm3):
| Path on master | Path on non-master |
|---|---|
/etc/ceph/ceph.client.admin.keyring |
/etc/ceph/ceph.client.admin.keyring |
/etc/ceph/ceph.conf |
/etc/ceph/ceph.conf |
/etc/ceph/ceph.pub |
/root/.ssh/authorized_keys |
Please replace root user with the username that can access the machine via ssh!!!
Please replace REMOTE_SSH_USER_PASSWORD with the password of the user that can access the machine via ssh!!!
!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:
sudo cat /etc/ceph/ceph.pub | ssh root@192.168.0.101 "tee /tmp/authorized_keys && echo 'REMOTE_SSH_USER_PASSWORD' | sudo -S sh -c 'mkdir -p /root/.ssh && cat /tmp/authorized_keys | tee -a /root/.ssh/authorized_keys > /dev/null && rm /tmp/authorized_keys'"
sudo cat /etc/ceph/ceph.pub | ssh root@192.168.0.102 "tee /tmp/authorized_keys && echo 'REMOTE_SSH_USER_PASSWORD' | sudo -S sh -c 'mkdir -p /root/.ssh && cat /tmp/authorized_keys | tee -a /root/.ssh/authorized_keys > /dev/null && rm /tmp/authorized_keys'"
sudo ceph orch host add dockerswm2 192.168.0.101 --labels _admin
sudo ceph orch host add dockerswm3 192.168.0.102 --labels _admin
Now we add OSDs to our ceph cluster.
!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:
sudo ceph orch apply osd --all-available-devices
This will identify any unpartitioned, unmounted disks attached to each participating node, and configure these disks as OSDs.
Now we create a cephfs volume in our cluster by running the following command:
!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:
sudo ceph fs volume create data
You can watch the progress by running sudo ceph fs ls (to see the fs is configured), and sudo ceph -s to wait for HEALTH_OK.
At last we mount the CephFS volume under /var/data on ervery docker swarm node:
The following commands are repeated for the following systems (<span style="color:#60A917">dockerswm1</span>; <span style="color:#0050EF">dockerswm2</span>; <span style="color:#A20025">dockerswm3</span>):
sudo mkdir /var/data
sudo sh -c 'echo "
# Mount cephfs volume
admin@.data=/ /var/data ceph mon_addr=192.168.1.100:6789/192.168.1.101:6789/192.168.0.102:6789,noatime,_netdev 0 0" >> /etc/fstab'
sudo mount -a
sudo systemctl daemon-reload
You can now create new files in the directory /var/data, which will then be synchronized to all machines.
3. Install Portainer
Portainer is a container for managing Docker containers via a graphical web interface.
Create a folder for the Portainer container (persistend data) (container volume):
!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:
sudo mkdir /var/data/portainer
Now a YAML file (docker-compose) is created to start the Portainer container in the Swarm environment. Please use nano to create the file portainer.yaml file and fill it with the code contained in the following code block and save it.
!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:
nano portainer.yaml
Insert the following content into the nano editor, save the file and close the nano editor:
version: '3.2'
services:
agent:
image: portainer/agent:2.21.1
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/volumes:/var/lib/docker/volumes
networks:
- agent_network
deploy:
mode: global
placement:
constraints: [node.platform.os == linux]
portainer:
image: portainer/portainer-ee:2.21.1
command: -H tcp://tasks.agent:9001 --tlsskipverify
ports:
- "9443:9443"
- "9000:9000"
- "8000:8000"
volumes:
- /var/data/portainer:/data
networks:
- agent_network
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
networks:
agent_network:
driver: overlay
attachable: true
4. Shutdown docker node where the portainer_portainer container is currently running
!!! Only on machine <span style="color:#60A917">dockerswm1</span> !!!:
sudo shutdown now
5. Portainer stack is restarting the portainer_portainer on one of the two remaining docker nodes but this error is appearing:
Command I used to log the events of the container:
sudo docker service logs portainer_portainer -f
Output:
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:98 > failed opening store | error="invalid database"
Portainer logs or screenshots
Command I used to log the events of the container:
sudo docker service logs portainer_portainer -f
Output:
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer_portainer.1.qv1leusuuuig@dockerswm2 | 2024/09/13 09:11PM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:98 > failed opening store | error="invalid database"
Portainer version
2.21.1
Portainer Edition
Business Edition (EE)
Platform and Version
Docker Swarm 27.2.1
OS and Architecture
Ubuntu 24.04
Browser
Firefox 130.0
What command did you use to deploy Portainer?
version: '3.2'
services:
agent:
image: portainer/agent:2.21.1
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/volumes:/var/lib/docker/volumes
networks:
- agent_network
deploy:
mode: global
placement:
constraints: [node.platform.os == linux]
portainer:
image: portainer/portainer-ee:2.21.1
command: -H tcp://tasks.agent:9001 --tlsskipverify
ports:
- "9443:9443"
- "9000:9000"
- "8000:8000"
volumes:
- /var/data/portainer:/data
networks:
- agent_network
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
networks:
agent_network:
driver: overlay
attachable: true
Additional Information
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 8
- Comments: 45
Sorry for my late response. @ibaraki-douji, I have now implemented your tips in my cluster:
(I have setup my cluter from the ground up)
Repeated all the steps below “1. Build Docker Swarm” from my initial bug report.
Put the following content in the config file (repeated on each node):
sudo apt install fuse-overlayfs(repeated on each node)rebooteach node.Verified on each node that the fuse-overlayfs drivers are used:
Output on is node is:
fuse-overlayfsRepeated all the steps below “2. Setup ceph storage cluster” from my initial bug report.
Added your suggested mount options in the
/etc/fstab(repeated on each node):Set ceph parameter
fuse_disable_pagecacheto true (ONLY ON ONE NODE):This commmand need a client name (in my example it is called
client). Is this client name correct ??? My three node have the following names (dockerswm1, dockerswm2 and dockerswm3)Check if parameter is set (repeated on each node):
Output on is node is:
client advanced fuse_disable_pagecache trueRepeated all the steps below “3. Install Portainer” from my initial bug report.
Unfortunately I get the same error with the portainer database after I shutdown the node where the docker service
portainer_portainerwas running:`sudo docker service logs portainer_portainer -f
I was doing the same thing until Gluster just didnt cut it anymore and i moved to Ceph. for what its worth, while this potential bug is evaluated, I moved Portainer to a raspberry pi I use for critical base services like DNS and just connected it using the Agent instead. I’d imagine you could easily run a copy of portainer on any old thing or even your primary workstation and tie it back using the agent temporarily. then once this is resolved we can all migrate our configs and portainer volumes back to the swarm.
I find it odd that this database in particular is having this issue on Ceph given that my postgres and sqlite db’s are having no issues.
I have portainer running in an Ubuntu 24.04 Virtual Machine. Ceph version is 19.2.0.
I have switched my portainer data path to a local folder within the VM and … SURPRISE … the issue is the same! It seems that this doesn’t even have anything to do with ceph, a very simple deploy using this:
sudo docker run -d --name portainer --restart=always -p 8000:8000 -p 9000:9000 -p 9443:9443 -v /var/run/docker.sock:/var/run/docker.sock -v /home/main/portainer:/data portainer/portainer-ce:latestresults in a setup that will compromise its own database. What is going on here?
same issue here; other db applications on cephfs seem to be unaffected e.g. slite3 or mysql. pve-cluster with 3 nodes and ceph. docker 27.3.1, portainer CE 2.21.4, ubuntu 24.04, ceph 18.2.4 will give these “mitigations” a shot.
my compose also looks a bit different:
edit: the flags etc. did not help.
i noticed i can provoke the crippled database by just creating a backup via gui.
after updating the service:
i have basically the exact same setup but with glusterfs as shared-fs backing the swarm, there the portainer-stuff works just fine. i wanted to migrate to ceph because a thousand other things dont work well.
unfortunately spoke to soon. after a reboot of the cluster it’s failing again. it seems to be related to file locking but im not certain yet.
My solution was to move away from cephfs for storing the database and instead have the container exclusively mount an RBD volume formatted to ext4. Running super stable throughout updates, upgrades, reboots, and host switching for months now. So its definitely some kind of compatibility issue with cephfs - which isn’t surprising. Clustered filesystems are complicated and not really intended for complicated workloads like databases.
I’ve got the same problem (Ubuntu 24.04 and Ceph 19.2 on Proxmox). Tried several inputs from this thread but nothing worked. My current solution is to use a CIFS volume mapped to a high available samba server (the storage is needed for the windows container hosts).
Hello @SuitDeer ,
Thank you for your detailed post. I’m working with the same setup as many here:
Shared files via CephFS but I got the same issue as you described…
Did you find a solution / workaround?
It’s my first time setting up a swarm cluster. For the moment it’s in a LAB / TEST environnement but I very feel unconfident about using Portainer in production… Even more after seeing that you’ve reported this issue back in September 2024 and we are in January 2025 and it’s not resolved yet…
Thank you!
Today I have build my docker swarm cluster from the ground up again. Now I am using the following tutorial: https://www.youtube.com/watch?v=nmas_zbcMeU
But with microceph (ubuntu) I get the same error 🤔
adding sync to the fstab line seems to have worked. I was able to shutdown and restart the container without encountering the db issue in the logs
hmm, strange, do you have removed the old invalid database after the fuse change ?
also are you using some options for your mount ?? if not try to add some options to remove all the async R/W https://docs.ceph.com/en/reef/man/8/mount.ceph/