argo-cd: Cannot connect to Private Gitlab Repo

Checklist:

  • I’ve searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
  • I’ve included steps to reproduce the bug.
  • I’ve pasted the output of argocd version.

Describe the bug

I have successfully connected to my private repo hosted on gitlab using the UI. The UI confirms successful connection. image

When trying to create an app I receive the following error

Unable to create application: application spec is invalid: 
InvalidSpecError: Unable to get app details: rpc error: code = Internal desc = Failed to fetch git 
repo: `git fetch origin --tags --force` failed exit status 128: 
Failed to add the ECDSA host key for IP address 'XX.XXX.XXX.XX' to the list of known hosts (/app/config/ssh/ssh_known_hosts). 
Load key "/dev/shm/568617259": invalid format git@gitlab.com: Permission denied (publickey). fatal: Could not read from remote repository. 
Please make sure you have the correct access rights and the repository exists.

Expected behavior

To successfully clone repo

Version

argocd: v1.3.6+89be1c9
  BuildDate: 2019-12-10T22:48:19Z
  GitCommit: 89be1c9ce6db0f727c81277c1cfdfb1e385bf248
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.3.6+89be1c9
  BuildDate: 2019-12-10T22:47:48Z
  GitCommit: 89be1c9ce6db0f727c81277c1cfdfb1e385bf248
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: Version: {Version:kustomize/v3.2.1 GitCommit:d89b448c745937f0cf1936162f26a5aac688f840 BuildDate:2019-09-27T00:10:52Z GoOs:linux GoArch:amd64}
  Helm Version: v2.15.2
  Kubectl Version: v1.14.0

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 9
  • Comments: 36 (15 by maintainers)

Commits related to this issue

Most upvoted comments

So just for the record: For any version not having the fix from #3064 included, the workaround is to a) either make sure that the SSH private key in the UI has a trailing new-line character or to add the private key via CLI.

I just had the same problem today. I was able to add the repository but couldn’t create an application. Turns out, Argo has a glitch when entering the private key via the UI…

Solution: I added my repository via the command line and it works now; I can then create an application successfully.

argocd repo add git@github.com:you/your-repo.git --ssh-private-key $HOME/.ssh/private_key

https://argoproj.github.io/argo-cd/user-guide/private-repositories/#ssh-private-key-credential

@jannfis Thank you for your response and sorry, I missed mentioning that I am receiving the same error message which has been originally posted on this issue:

Unable to create application: application spec is invalid: InvalidSpecError: Unable to get app details: rpc error: code = Internal desc = Failed to fetch git repo: git fetch origin --tags --force failed exit status 128: Failed to add the RSA host key for IP address ‘[repo server]’ to the list of known hosts (/app/config/ssh/ssh_known_hosts). Load key “/dev/shm/422544524”: invalid format git@[repo server]: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

To help with reproducing this specific issue, I’ve created two new repositories argocd-gh2890-1 and argocd-gh2890-2, created a 4096 bit RSA key using ssh-keygen and added the same public key to both repositories as a deployment key (= read-only repository access). I’ve then proceeded by pushing the same contents to both repositories and adding them in separate ways:

  1. argocd-gh2890-1 has been added through the UI, ensuring that no spaces are present before or after the private SSH key which I’ve copy-pasted from the terminal
  2. argocd-gh2890-2 has been added through the CLI using this command: argocd repo add git@[repo server]:ppmathis/argocd-gh2890-2.git --ssh-private-key-path argocd --name gh2890-2

So far the behavior does not differ and both repositories show up as successfully connected. When trying to add a new application, the outcome is different:

  1. argocd-gh2890-1 suffers from the issue I mentioned and throws the error message I quoted in the beginning of this issue. The path selector does not show any suggestions and adding the application fails.
  2. argocd-gh2890-2 works just fine. The path selector shows all recognized kustomize deployment files and adding the application works.

I continued by extracting the secrets into separate files and analyzed them using xxd and vimdiff:

kubectl -n argocd get secrets -o go-template --template '{{.data.sshPrivateKey}}' repo-argocd-gh2890-1-3746518547 | base64 -d > secret1
kubectl -n argocd get secrets -o go-template --template '{{.data.sshPrivateKey}}' repo-argocd-gh2890-2-163780834 | base64 -d > secret2
vimdiff <(xxd secret1) <(xxd secret2)

vimdiff of k8s secrets

The devil is in the detail as the screenshot clearly shows, as the secret added through the CLI (secret2 - right side) has a newline at the end whereas the key added through the WebUI does not. This also explains why the issue only occurs for some people when using the WebUI - when copying from a terminal (e.g. cat + manual selection), the newline usually does not get copied, however when copying from a graphical text editor the newline will be part of the clipboard.

To confirm the issue, I’ve added a third repository with the same contents named argocd-gh2890-3 with the same deployment key and added it through the UI once again, ensuring that the private key has a newline at the end - and everything works smoothly.

TL;DR: The SSH private key must have a single newline character at the end. While both cases (with and without) work just fine for the repository status check, the actual git fetch command fails without a trailing newline.

For the time being, can you please try to add the repository using a full qualified SSH URL, i.e. ssh://git@github.com/you/your-repo.git instead of git@github.com:you/your-repo.git and see if that functions as a workaround?

Indeed if I tcpdump on the host I see packets going to port 22 when I attempt to add the app.

I was able to fix this in Kubernetes by setting runAsUser to a valid userID that existed in the argo-cd-repo-server container:

       securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - all
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 65534