portainer: Portainer fails to deploy stack with GitOps polling, private repo, relative volumes

Before you start please confirm the following.

Yes, I’ve searched similar issues on GitHub.
Yes, I’ve checked whether this issue is covered in the Portainer documentation or knowledge base.

Problem Description

With a private repo, GitOps enabled, and relative volume mounts then Portainer will fail to copy the repo’s file most of the time when polling and attempting to deploy a new commit. Force pulling and redeploying works and populates the file, which are then deleted by the next polling.

Expected Behavior

The files are copied over and not erased.

Actual Behavior

The files aren’t copied over.

Steps to Reproduce

Create private repo
Enable GitOps, polling, with credentials
Do a force pull and redeploy, the files are there
Let Portainer pull the new commit via polling, the files are gone

Portainer logs or screenshots

No response

Portainer version

2.19.2

Portainer Edition

Business Edition (BE/EE) with 5NF / 3NF license

Platform and Version

Swarm 24.0.7

OS and Architecture

Mint 20.3 AMD64

Browser

No response

What command did you use to deploy Portainer?

https://downloads.portainer.io/ee2-19/portainer-agent-stack.yml

Additional Information

No response

About this issue

Original URL
State: open
Created 7 months ago
Reactions: 4
Comments: 18 (3 by maintainers)

Most upvoted comments

Here’s the problem:

https://github.com/portainer/portainer/blob/develop/api/git/types/types.go#L25C1-L33C2

type GitAuthentication struct {
	Username string
	Password string
	// Git credentials identifier when the value is not 0
	// When the value is 0, Username and Password are set without using saved credential
	// This is introduced since 2.15.0
	GitCredentialID int `example:"0"`
}

and when building the unpacker command it calls appendGitAuthIfNeeded

https://github.com/portainer/portainer/blob/develop/api/stacks/deployments/compose_unpacker_cmd_builder.go#L185-L190

func appendGitAuthIfNeeded(cmd []string, stack *portainer.Stack) []string {
	if stack.GitConfig.Authentication != nil && len(stack.GitConfig.Authentication.Password) != 0 {
		cmd = append(cmd, "-u", stack.GitConfig.Authentication.Username, "-p", stack.GitConfig.Authentication.Password)
	}
	return cmd
}

here it only checks if Username or Password is set, otherwise it skips authentication completely. However if you have stored credentials, the Username and Password will be blank and only the GitCredentialID is set. I guess it works when creating the stack because the Username and Password is set in the creation process, but won’t get set in the reconciliation when pulling updates.

appendGitAuthIfNeeded needs to resolve the GitCredentialID into Username and Password and append it to the unpacker command options.

sled on Mar 14, 2024

Webhook also returns an error:

{
    "message": "Failed to update the stack",
    "details": "failed to deploy a docker compose stack 8: an error occurred while running unpacker container with exit code 255: authentication required"
}

ethrgeist on Jan 19, 2024

I have a the same behavior.

Polling is failing to redeploy a updated compose file from git.

The frontend notification is could not get the contents of the file 'docker-compose.yml'

Log shows:

2024/01/19 01:03PM INF github.com/portainer/portainer-ee/api/stacks/deployments/deployer_remote.go:207 > Stack deployment output | output="{"level":"info","repository":"https://github.com/example/private-repo","composePath":["docker-compose.yml"],"destination":"/opt/portainer-compose-unpacker","env":[],"skipTLSVerify":false,"time":1705669398,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:31","message":"Deploying Compose stack from Git repository"}
{"level":"info","time":1705669398,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:300","message":"Docker login ghcr.io successed"}
{"level":"info","time":1705669399,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:300","message":"Docker login docker.io successed"}
{"level":"info","user":"gituseraccount","time":1705669399,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:42","message":"Using Git authentication"}
{"level":"info","directory":"/opt/portainer-compose-unpacker","time":1705669399,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:57","message":"Checking the file system..."}
{"level":"info","directory":"/opt/portainer-compose-unpacker/stacks/stackname","time":1705669399,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:83","message":"Creating target destination directory on disk"}
{"level":"info","repository":"https://github.com/example/private-repo","path":"/opt/portainer-compose-unpacker/stacks/stackname/private-repo","url":"https://github.com/example/private-repo","depth":1,"time":1705669399,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:98","message":"Cloning git repository"}
{"level":"info","composeFilePaths":["/opt/portainer-compose-unpacker/stacks/stackname/private-repo/docker-compose.yml"],"workingDirectory":"/opt/portainer-compose-unpacker/stacks/stackname/private-repo","projectName":"stackname","time":1705669400,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:126","message":"Deploying Compose stack"}
{"level":"info","time":1705669418,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:144","message":"Compose stack deployment complete"}
{"level":"info","time":1705669418,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:331","message":"Docker logout ghcr.io successed"}
{"level":"info","time":1705669418,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:331","message":"Docker logout docker.io successed"}
"
2024/01/19 01:06PM INF github.com/portainer/portainer-ee/api/stacks/deployments/deployer_remote.go:207 > Stack deployment output | output="{"level":"info","repository":"https://github.com/example/private-repo","composePath":["docker-compose.yml"],"destination":"/opt/portainer-compose-unpacker","env":[],"skipTLSVerify":false,"time":1705669586,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:31","message":"Deploying Compose stack from Git repository"}
{"level":"info","time":1705669587,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:300","message":"Docker login ghcr.io successed"}
{"level":"info","time":1705669588,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:300","message":"Docker login docker.io successed"}
{"level":"info","directory":"/opt/portainer-compose-unpacker","time":1705669588,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:57","message":"Checking the file system..."}
{"level":"info","directory":"/opt/portainer-compose-unpacker/stacks/stackname","time":1705669588,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:83","message":"Creating target destination directory on disk"}
{"level":"info","repository":"https://github.com/example/private-repo","path":"/opt/portainer-compose-unpacker/stacks/stackname/private-repo","url":"https://github.com/example/private-repo","depth":1,"time":1705669588,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:98","message":"Cloning git repository"}
{"level":"error","error":"authentication required","time":1705669588,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:104","message":"Failed to clone Git repository"}
{"level":"info","time":1705669588,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:331","message":"Docker logout ghcr.io successed"}
{"level":"info","time":1705669588,"caller":"/home/runner/work/compose-unpacker/compose-unpacker/deploy.go:331","message":"Docker logout docker.io successed"}
"
2024/01/19 01:06PM ERR github.com/portainer/portainer-ee/api/scheduler/scheduler.go:113 > job returned an error, it will be rescheduled | error="failed to deploy a docker compose stack 8: an error occurred while running unpacker container with exit code 255: authentication required"

Using the pull and redeploy button updates the containers, but it does not spawn the files on the filesystem. The files that were cloned when first creating the stack are also deleted after the first poll.

ethrgeist on Jan 19, 2024

#10630 might be playing a part here - we’ve just released 2.19.3 which should fix this, so while I don’t think it solves all your issues it might help with some. But there is some useful detail here that I’ll pass on to the team, thanks.

jamescarppe on Nov 21, 2023