portainer: SWARM: Can not edit any service after deployment .> Showstopper

Before you start please confirm the following.

Problem Description

I am running a Docker Swarm on 5 nodes of Oracle Linux 8.

Using Portainer EE 2.19.2 everytime i try to edit a parameter of one of my services i see a popup in the right upper corner telling me:

invalid CredentialSpec: must either provide "file", "registry", or "config" for credential spec

So defacto i can not edit my services from Portainer any more.

Expected Behavior

Let me please edit my services again.

Actual Behavior

Can not alter a parameter of a running service on my Docker Swarm on Linux.

Steps to Reproduce

  1. goto any service
  2. open for editing
  3. eg go to environment vars and try to add a test veriable
  4. -> when saving the popup appears

Portainer logs or screenshots

No response

Portainer version

2.19.2

Portainer Edition

Business Edition (BE/EE) with 5NF / 3NF license

Platform and Version

Docker Swarm 24.0.7, build afdd53b

OS and Architecture

Oracle Linux 8 on arm64

Browser

Chrome latest

What command did you use to deploy Portainer?

deploy and updating

docker pull portainer/portainer-ee:latest                                                                                                                            
docker service update --image portainer/portainer-ee:latest --publish-add 9443:9443 --force portainer_portainer

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Reactions: 19
  • Comments: 30 (3 by maintainers)

Most upvoted comments

Tested. Fix confirmed. TY!

Hi all - as noted above we have a fix in testing for release as part of 2.19.4 - I don’t have an ETA on release of this patch as of yet but we’re working to get this out as soon as possible as we know this bug has been disruptive for a number of you.

Finally, I found a solution to roll back. In your volume /data/backups/EE, there is a database backup. You can copy that to /data/ and replace portainer.db. Restart and change the image to 2.19.1, and it works for me~

Same for our prod environment 😞 Is there a mechanism to migrate down the database ?

Update

It workes well with the v2.19.0 (No need to downgrade the agents)

docker service update --image portainer/portainer-ce:2.19.0 [SERVICE_NAME]

Good news, 2.19.4 is now out, and fixes this issue. We have noticed on occasion the error might still show up immediately after updating, but this appears to be a caching issue so make sure you clear your browser cache if this happens.

Workaround I found that doesn’t involve downgrading, assuming you’re using Chrome and you have access to a terminal with the curl command. The process should be similar for Firefox/etc as long as your browser has a DevTools equivalent.

  1. Make your changes to the service form as normal but do not submit them
  2. Open DevTools and change to the Network tab
  3. Submit the form and find the failed request, should be a 400 response code.
  4. Right click the request, select Copy > Copy as cURL
  5. Paste the cURL command into your favourite text editor
  6. In the --data-raw argument at the end, find the text "Privileges":{"CredentialSpec":{}, and remove the part "CredentialSpec": {},
  7. Copy the new, modified, curl command
  8. Paste into a terminal and run the command.
  9. Ideally you should get this response: {"Warnings":null}
  10. Refresh your services page and verify the change was made.

I upgraded from 2.19.1-CE to 2.19.3-CE and started to receive the :invalid CredentialSpec: must either provide “file”, “registry”, or “config” for credential spec"

Same for our prod environment 😞 Is there a mechanism to migrate down the database ?

Update

It workes well with the v2.19.0 (No need to downgrade the agents)

docker service update --image portainer/portainer-ce:2.19.0 [SERVICE_NAME]

While waiting for version 2.19.4, I fixed it this way. Downgrading from 2.19.3 to version 2.19.0 (not to version 2.19.1)

Also wonder why there ist no hotfix for this, its quite a major showstopping bug…

Same here:

invalid CredentialSpec: must either provide “file”, “registry”, or “config” for credential spec

Tried downgrading with the same error:

FTL github.com/portainer/portainer/api/cmd/portainer/main.go:403 > The database schema version does not align with the server version. Please consider reverting to the previous server version or addressing the database migration issue.

In the meantime a new patch version got released 2 days ago: 2.19.3

Hoped it fixes the bug, but nooope.

grafik

🤔