argo-cd: Unable to create application from a connected Azure DevOps server repository

Checklist:

  • I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I’ve included steps to reproduce the bug.
  • I’ve pasted the output of argocd version.

Describe the bug

After not being able to connect a private Azure DevOps server repository to ArgoCD via username and password (with error returned “Unable to connect HTTPS repository: authentication required)”, I have successfully connected the repo via Personal Access Token (PAT):

image

When I try to create an application using the repository, I get the following error:

Unable to create application: application spec is invalid: InvalidSpecError: Unable to get app details: rpc error: code = Internal desc = Failed to fetch 493dd952d54b8817daa972adc056d32a1f31231a: git fetch origin --tags --force failed exit status 128: fatal: Authentication failed for '<redacted>/_git/argocd-apps/'

PAT used has Full Access scope and works with git operations to the repo, which was tested locally with git clone and git fetch origin --tags --force which is reported in the ArgoCD error message.

My guess is that this is related to Azure DevOps not allowing PAT authentication with a non-empty username (tested locally with git), while ArgoCD requires username to be a non-empty string when adding repository credentials. ArgoCD is probably passing this username to Azure DevOps which results in the mentioned error. Related issue: #6098

To Reproduce

  • Create a repository on Azure Devops server
  • Create an Azure DevOps server Personal Access Token (PAT) with Full Access scope
  • Add the repository to ArgoCD via HTTPS using the PAT
  • Try to create an ArgoCD application using the added repository

Expected behavior

ArgoCD application is successfully created.

Screenshots

image

Version

argocd-server: v2.0.0+f5119c0
  BuildDate: 2021-04-07T06:00:33Z
  GitCommit: f5119c06686399134b3f296d44445bcdbc778d42
  GitTreeState: clean
  GoVersion: go1.16
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.9.4 2021-02-09T19:22:10Z
  Helm Version: v3.5.1+g32c2223
  Kubectl Version: v0.20.4
  Jsonnet Version: v0.17.0

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 9
  • Comments: 17 (5 by maintainers)

Most upvoted comments

That’s interesting - it seems that this is an Azure DevOps Services feature, since I don’t have that option in Azure DevOps Server 2020:

image

I have created the PAT as described here and no username was given. Here are some findings:

  • git v2.9.2.windows.1: I can git clone the repo using empty username and PAT (in clear text, not base64 encoded). Non-empty username is a no go.
  • git v2.18.4: I am unable to git clone the repository using any combination of non-base64 encoded username and PAT (tried with empty username, my AzDO username with the domain name, my AzDO username without the domain name, random string, event the token name). The only way to clone is as described here.

So it’s seems the behavior also depends on the git version (possibly also on the OS?). I’d say that currently the safest way to use AzDO Server with ArgoCD is by adding the repo via SSH.

@nilic This error is from github directly. While accessing a repo, github returns fatal: Authentication failed error if username is not provided.

$ git clone https://github.com/username/Private-Repo-Test.git
Cloning into 'Private-Repo-Test'...
Username for 'https://github.com':
Password for 'https://github.com': <personal-access-token>
remote: Repository not found.
fatal: Authentication failed for 'https://github.com/kshamajain99/Private-Repo-Test.git/'

$ git clone https://github.com/username/Private-Repo-Test.git
Cloning into 'Private-Repo-Test'...
Username for 'https://github.com': fakeusername
Password for 'https://fakeusername@github.com': <personal-access-token>
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 9 (delta 0), reused 6 (delta 0), pack-reused 0
Receiving objects: 100% (9/9), done.

For Azure DevOps Server 2020 Update 1.1 this workaround is needed: https://github.com/argoproj/argo-cd/issues/6812#issuecomment-1166957798, just replace ssh.dev.azure.com with the name of your Azure DevOps server and remove line User git.

Azure DevOps Server 2022 RC2 adds support for additional signature algorithms besides ssh-rsa so hopefully this workaround won’t be necessary after upgrading to the new version - Azure DevOps Server 2022 RC2 Release Notes

I have tested ArgoCD v2.5.0+a765c14 with Azure DevOps Server 2022 RC2 and can confirm that the ssh-rsa workaround is no longer needed.

Did anyone find any workarounds?

@luber I think using ssh. However, expect problems.

Firstly, with on-prem hosting. You’re going to run into: https://argo-cd.readthedocs.io/en/release-2.0/user-guide/private-repositories/#unknown-ssh-hosts

Fix for me by doing:

ssh-keyscan your-server-host-name | argocd cert add-ssh --batch

Then, because ArgoCD removed support for SHA-1 signature hash algorithm and Azure Devops still seems to be on SHA-1 [1] [2]

Think you may need to modify argocd install to support signature hash algorithm AzDO supports sort of like this https://github.com/argoproj/argo-cd/issues/6812#issuecomment-1073918267

I got to this point and we decided to start migrating off Azure Devops to Github instead because we doubt Microsoft is going to continue to support Azure Devops on the same level as Github in the future.

@nilic we have successfully used azure devops with argo by creating the PAT from the clone window. There it specifies the exact username to use. This does work, random username does not:

image