argo-cd: Unable to create application from a connected Azure DevOps server repository
Checklist:
- I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
- I’ve included steps to reproduce the bug.
- I’ve pasted the output of
argocd version
.
Describe the bug
After not being able to connect a private Azure DevOps server repository to ArgoCD via username and password (with error returned “Unable to connect HTTPS repository: authentication required)”, I have successfully connected the repo via Personal Access Token (PAT):
When I try to create an application using the repository, I get the following error:
Unable to create application: application spec is invalid: InvalidSpecError: Unable to get app details: rpc error: code = Internal desc = Failed to fetch 493dd952d54b8817daa972adc056d32a1f31231a:
git fetch origin --tags --force failed exit status 128: fatal: Authentication failed for '<redacted>/_git/argocd-apps/'
PAT used has Full Access scope and works with git
operations to the repo, which was tested locally with git clone
and git fetch origin --tags --force
which is reported in the ArgoCD error message.
My guess is that this is related to Azure DevOps not allowing PAT authentication with a non-empty username (tested locally with git
), while ArgoCD requires username to be a non-empty string when adding repository credentials. ArgoCD is probably passing this username to Azure DevOps which results in the mentioned error. Related issue: #6098
To Reproduce
- Create a repository on Azure Devops server
- Create an Azure DevOps server Personal Access Token (PAT) with Full Access scope
- Add the repository to ArgoCD via HTTPS using the PAT
- Try to create an ArgoCD application using the added repository
Expected behavior
ArgoCD application is successfully created.
Screenshots
Version
argocd-server: v2.0.0+f5119c0
BuildDate: 2021-04-07T06:00:33Z
GitCommit: f5119c06686399134b3f296d44445bcdbc778d42
GitTreeState: clean
GoVersion: go1.16
Compiler: gc
Platform: linux/amd64
Ksonnet Version: v0.13.1
Kustomize Version: v3.9.4 2021-02-09T19:22:10Z
Helm Version: v3.5.1+g32c2223
Kubectl Version: v0.20.4
Jsonnet Version: v0.17.0
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 9
- Comments: 17 (5 by maintainers)
That’s interesting - it seems that this is an Azure DevOps Services feature, since I don’t have that option in Azure DevOps Server 2020:
I have created the PAT as described here and no username was given. Here are some findings:
git v2.9.2.windows.1
: I cangit clone
the repo using empty username and PAT (in clear text, not base64 encoded). Non-empty username is a no go.git v2.18.4
: I am unable togit clone
the repository using any combination of non-base64 encoded username and PAT (tried with empty username, my AzDO username with the domain name, my AzDO username without the domain name, random string, event the token name). The only way to clone is as described here.So it’s seems the behavior also depends on the
git
version (possibly also on the OS?). I’d say that currently the safest way to use AzDO Server with ArgoCD is by adding the repo via SSH.@nilic This error is from github directly. While accessing a repo, github returns
fatal: Authentication failed
error if username is not provided.For Azure DevOps Server 2020 Update 1.1 this workaround is needed: https://github.com/argoproj/argo-cd/issues/6812#issuecomment-1166957798, just replace
ssh.dev.azure.com
with the name of your Azure DevOps server and remove lineUser git
.Azure DevOps Server 2022 RC2 adds support for additional signature algorithms besides
ssh-rsa
so hopefully this workaround won’t be necessary after upgrading to the new version - Azure DevOps Server 2022 RC2 Release NotesI have tested ArgoCD v2.5.0+a765c14 with Azure DevOps Server 2022 RC2 and can confirm that the
ssh-rsa
workaround is no longer needed.@luber I think using ssh. However, expect problems.
Firstly, with on-prem hosting. You’re going to run into: https://argo-cd.readthedocs.io/en/release-2.0/user-guide/private-repositories/#unknown-ssh-hosts
Fix for me by doing:
Then, because ArgoCD removed support for SHA-1 signature hash algorithm and Azure Devops still seems to be on SHA-1 [1] [2]
Think you may need to modify argocd install to support signature hash algorithm AzDO supports sort of like this https://github.com/argoproj/argo-cd/issues/6812#issuecomment-1073918267
I got to this point and we decided to start migrating off Azure Devops to Github instead because we doubt Microsoft is going to continue to support Azure Devops on the same level as Github in the future.
@nilic we have successfully used azure devops with argo by creating the PAT from the clone window. There it specifies the exact username to use. This does work, random username does not: