portainer: Migration of db fails when upgrading to 2.19.4 when using network file system for data volume

Before you start please confirm the following.

Problem Description

Error when deploying 2.19.4 CE docker image. Failed migration | error=“while backing up database: read /data/portainer.db: permission denied”

I am able to rollback to the image 2.19.4 CE and access the db fine and container starts. I had this issue previously here: https://github.com/portainer/portainer/issues/7144

Expected Behavior

Able to apply latest version.

Actual Behavior

Error when migrating.

Steps to Reproduce

Pulled new image (2.19.4) using docker-compose and started container.

Portainer logs or screenshots

2023/12/07 12:45AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:369 > encryption key file not present | filename=portainer
2023/12/07 12:45AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
2023/12/07 12:45AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/12/07 12:45AM INF github.com/portainer/portainer/api/datastore/backup.go:16 > Backing up database | from=/data/portainer.db to=/data/backups/portainer.db.bak
2023/12/07 12:45AM WRN github.com/portainer/portainer/api/datastore/backup.go:19 > failed to create backup file | error="read /data/portainer.db: permission denied"
2023/12/07 12:45AM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:142 > failed migration | error="while backing up database: read /data/portainer.db: permission denied" stack_trace=[{"func":"(*Store).MigrateData","line":"45","source":"migrate_data.go"},{"func":"initDataStore","line":"140","source":"main.go"},{"func":"buildServer","line":"395","source":"main.go"},{"func":"main","line":"640","source":"main.go"},{"func":"main","line":"250","source":"proc.go"},{"func":"goexit","line":"1598","source":"asm_amd64.s"}]

Portainer version

2.19.4

Portainer Edition

Community Edition (CE)

Platform and Version

24.0.5, build ced0996

OS and Architecture

Ubuntu 22.04 AMD64

Browser

No response

What command did you use to deploy Portainer?

services:
  portainer:
    container_name: portainer
    image: portainer/portainer-ce:latest
    ports:
      - "$PORTAINER_PORT:9000/tcp"
    volumes:
      - portainer_data:/data

volumes:
  portainer_data:
    driver_opts:
      o: "username=xyz,password=xyz"
      type: "cifs"
      device: \\10.1.1.1\portainer\data

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 7 months ago
  • Reactions: 3
  • Comments: 17 (4 by maintainers)

Most upvoted comments

We have now pushed a new version of the 2.19.4 image that fixes this issue. As part of this we removed the 2.19.4 image temporarily and re-tagged latest to 2.19.3 while we worked on it, and it looks like this caused other issues in the interim. Thanks for your patience while this got sorted out!

I fixed the missing 2.19.4 tag by doing a database rollback to 2.19.3:

  • In the stack file, revert Docker tag to 2.19.3
  • cp backups/portainer.db.bak portainer.db
  • Restart Portainer

I don’t know what’s going on here, but removing an existing Docker tag seems like bad practice to me.

Having similar trouble. Can’t fix, because Docker image for 2.19.4 seems to be gone: https://hub.docker.com/r/portainer/portainer-ce/tags https://hub.docker.com/r/portainer/portainer-ee/tags

2.19.3 cannot be used because database is already on 2.19.4.

I don’t have a copy of the DB to rollback to 2.19.3… should I just wait for the new 2.19.4 to pop out again? (using BE edition)

The new version is out, as I mentioned above. Re-pull the image from Docker Hub and you’re good to redeploy.

Instead of :latest you can explicitly specify a version like portainer/portainer-ce:2.19.4 if needed. I also recommend stopping and removing the container to ensure the new image is used. As long as your portainer_data volume isn’t removed you’ll retain your configuration.

docker stop portainer
docker rm portainer
docker pull portainer/portainer-ce:latest
docker run ...

Just wanted to update that completely removing the portainer container and recreating using a docker compose file resolved my issue.

docker rm portainer
cd /path/to/portainer/compose/file
docker compose up -d

The issue just got corrected it seems. Thank you very much for your quick response.