argo-cd: Cannot connect to Private Gitlab Repo
Checklist:
- I’ve searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
- I’ve included steps to reproduce the bug.
- I’ve pasted the output of
argocd version
.
Describe the bug
I have successfully connected to my private repo hosted on gitlab using the UI. The UI confirms successful connection.
When trying to create an app I receive the following error
Unable to create application: application spec is invalid:
InvalidSpecError: Unable to get app details: rpc error: code = Internal desc = Failed to fetch git
repo: `git fetch origin --tags --force` failed exit status 128:
Failed to add the ECDSA host key for IP address 'XX.XXX.XXX.XX' to the list of known hosts (/app/config/ssh/ssh_known_hosts).
Load key "/dev/shm/568617259": invalid format git@gitlab.com: Permission denied (publickey). fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
Expected behavior
To successfully clone repo
Version
argocd: v1.3.6+89be1c9
BuildDate: 2019-12-10T22:48:19Z
GitCommit: 89be1c9ce6db0f727c81277c1cfdfb1e385bf248
GitTreeState: clean
GoVersion: go1.12.6
Compiler: gc
Platform: darwin/amd64
argocd-server: v1.3.6+89be1c9
BuildDate: 2019-12-10T22:47:48Z
GitCommit: 89be1c9ce6db0f727c81277c1cfdfb1e385bf248
GitTreeState: clean
GoVersion: go1.12.6
Compiler: gc
Platform: linux/amd64
Ksonnet Version: v0.13.1
Kustomize Version: Version: {Version:kustomize/v3.2.1 GitCommit:d89b448c745937f0cf1936162f26a5aac688f840 BuildDate:2019-09-27T00:10:52Z GoOs:linux GoArch:amd64}
Helm Version: v2.15.2
Kubectl Version: v1.14.0
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 9
- Comments: 36 (15 by maintainers)
So just for the record: For any version not having the fix from #3064 included, the workaround is to a) either make sure that the SSH private key in the UI has a trailing new-line character or to add the private key via CLI.
I just had the same problem today. I was able to add the repository but couldn’t create an application. Turns out, Argo has a glitch when entering the private key via the UI…
Solution: I added my repository via the command line and it works now; I can then create an application successfully.
https://argoproj.github.io/argo-cd/user-guide/private-repositories/#ssh-private-key-credential
@jannfis Thank you for your response and sorry, I missed mentioning that I am receiving the same error message which has been originally posted on this issue:
To help with reproducing this specific issue, I’ve created two new repositories
argocd-gh2890-1
andargocd-gh2890-2
, created a 4096 bit RSA key usingssh-keygen
and added the same public key to both repositories as a deployment key (= read-only repository access). I’ve then proceeded by pushing the same contents to both repositories and adding them in separate ways:argocd-gh2890-1
has been added through the UI, ensuring that no spaces are present before or after the private SSH key which I’ve copy-pasted from the terminalargocd-gh2890-2
has been added through the CLI using this command:argocd repo add git@[repo server]:ppmathis/argocd-gh2890-2.git --ssh-private-key-path argocd --name gh2890-2
So far the behavior does not differ and both repositories show up as successfully connected. When trying to add a new application, the outcome is different:
argocd-gh2890-1
suffers from the issue I mentioned and throws the error message I quoted in the beginning of this issue. The path selector does not show any suggestions and adding the application fails.argocd-gh2890-2
works just fine. The path selector shows all recognized kustomize deployment files and adding the application works.I continued by extracting the secrets into separate files and analyzed them using xxd and vimdiff:
The devil is in the detail as the screenshot clearly shows, as the secret added through the CLI (
secret2
- right side) has a newline at the end whereas the key added through the WebUI does not. This also explains why the issue only occurs for some people when using the WebUI - when copying from a terminal (e.g.cat
+ manual selection), the newline usually does not get copied, however when copying from a graphical text editor the newline will be part of the clipboard.To confirm the issue, I’ve added a third repository with the same contents named
argocd-gh2890-3
with the same deployment key and added it through the UI once again, ensuring that the private key has a newline at the end - and everything works smoothly.TL;DR: The SSH private key must have a single newline character at the end. While both cases (with and without) work just fine for the repository status check, the actual
git fetch
command fails without a trailing newline.For the time being, can you please try to add the repository using a full qualified SSH URL, i.e.
ssh://git@github.com/you/your-repo.git
instead ofgit@github.com:you/your-repo.git
and see if that functions as a workaround?Indeed if I tcpdump on the host I see packets going to port 22 when I attempt to add the app.
I was able to fix this in Kubernetes by setting
runAsUser
to a valid userID that existed in the argo-cd-repo-server container: