argo-cd: argocd cluster add fails

Checklist:

I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I’ve included steps to reproduce the bug.
I’ve pasted the output of argocd version.

Describe the bug

argocd cluster add <cluster> fails.

To Reproduce

Install argocd in local docker desktop k8s, and login argocd by argocd login command.
minikube start. kubectl to minikube is no problem.
exec kubectl port-forward svc/argocd-server -n argocd 8080:443
exec argocd cluster add minikube --insecure . logs are below.

INFO[0000] ServiceAccount "argocd-manager" created in namespace "kube-system" 
INFO[0000] ClusterRole "argocd-manager-role" created    
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" created 
FATA[0001] rpc error: code = Unknown desc = Get "https://127.0.0.1:32768/version?timeout=32s": dial tcp 127.0.0.1:32768 connect: connection refused

argocd cluster add <k3d cluster running in local> also fails. The error message is the same.

Expected behavior

Add cluster succeeds.

Version

argocd: v1.7.1+da5fa74
  BuildDate: 2020-08-26T21:13:01Z
  GitCommit: da5fa74ca1bf9e025dcfe3c8cac083da1b7b45c8
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.7.2+c342d3f
  BuildDate: 2020-08-27T23:34:52Z
  GitCommit: c342d3fc9c9c9f0d1c18254b6ffa1e106984a76c
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.17.8

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (3 by maintainers)

Most upvoted comments

Workaround for kind, might work for other single node k8s solutions:

kubectl get endpoints

NAME         ENDPOINTS         AGE
kubernetes   172.19.0.2:6443   38m

Find the entry belonging to the cluster in your .kube/config, and change the server entry:

- cluster:
    certificate-authority-data: ...
    server: https://172.19.0.2:6443
  name: yourcontext

Verify that kubectl get pods is still working, then try argocd cluster add.

+30

norbertkeri on Feb 15, 2022

You can also use the --in-cluster flag

❯ argocd cluster add rancher-desktop --label environment=dev --insecure -y
INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated
FATA[0000] rpc error: code = Unknown desc = Get "https://127.0.0.1:6443/version?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused

❯ argocd cluster add rancher-desktop --label environment=dev --insecure --in-cluster -y --upsert
INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated
Cluster 'https://kubernetes.default.svc' added

EDIT: the above doesn’t work when it comes to deploying but allows the cluster to be added 😃

Kind does have the kind get kubeconfig --internal --name <cluster name> command.

+20

rumstead on Aug 30, 2022

You can also use the --in-cluster flag

❯ argocd cluster add rancher-desktop --label environment=dev --insecure -y
INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated
FATA[0000] rpc error: code = Unknown desc = Get "https://127.0.0.1:6443/version?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused

❯ argocd cluster add rancher-desktop --label environment=dev --insecure --in-cluster -y --upsert
INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated
Cluster 'https://kubernetes.default.svc' added

EDIT: the above doesn’t work when it comes to deploying but allows the cluster to be added 😃

Kind does have the kind get kubeconfig --internal --name <cluster name> command.

This works great with minikube. Thanks!

ranajoy-dutta on Sep 21, 2022

Hi, cluster add takes cluster API information from your K8s client configuration. Most likely, this is set to localhost. The error message you see comes from the argocd-server pod, who obviously cannot connect to any K8s API at 127.0.0.1:32768.

To solve this, modify the context in your ~/.kube/config to point to an IP reachable from within your Docker desktop K8s (and possibly reconfigure your K3s or minikube API server to listen not only on localhost.

jannfis on Aug 29, 2020

You can also use the --in-cluster flag

❯ argocd cluster add rancher-desktop --label environment=dev --insecure -y
INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated
FATA[0000] rpc error: code = Unknown desc = Get "https://127.0.0.1:6443/version?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused

❯ argocd cluster add rancher-desktop --label environment=dev --insecure --in-cluster -y --upsert
INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated
Cluster 'https://kubernetes.default.svc' added

EDIT: the above doesn’t work when it comes to deploying but allows the cluster to be added 😃 Kind does have the kind get kubeconfig --internal --name <cluster name> command.

This works great with minikube. Thanks!

Also worked with kind cluster

jaythamke on Nov 6, 2023

for those who are experiencing the issue while running kind clusters on mac, I have created this repository: https://github.com/akram/docker-argo-oc-kind

It builds an image with docker, argo and kubectl, and it documents how to change the ~/.kube/config to make clusters addable to argo

akram on Aug 16, 2023

I have tried to above work around, but none is working for me Executing on local cluster argocd cluster add kind-dev-cluster --insecure Error

WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `kind-dev-cluster` with full cluster level admin privileges. Do you want to continue [y/N]? y
INFO[0001] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0001] ClusterRole "argocd-manager-role" updated
INFO[0001] ClusterRoleBinding "argocd-manager-role-binding" updated
FATA[0002] rpc error: code = Unknown desc = Get "https://127.0.0.1:62826/version?timeout=32s": dial tcp 127.0.0.1:62826: connect: connection refused

we are able to connect https://127.0.0.1:62826/version?timeout=32s as this is not authenticate, here is the argocd server error

time="2022-01-13T08:07:10Z" level=info msg="received unary call /version.VersionService/Version" grpc.method=Version grpc.request.claims="{\"exp\":1642146356,\"iat\":1642059956,\"iss\":\"argocd\",\"jti\":\"92238083-20f4-4bda-81da-b9061edadbd8\",\"nbf\":1642059956,\"sub\":\"admin\"}" grpc.request.content= grpc.service=version.VersionService grpc.start_time="2022-01-13T08:07:10Z" span.kind=server system=grpc
time="2022-01-13T08:07:10Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.service=version.VersionService grpc.start_time="2022-01-13T08:07:10Z" grpc.time_ms=5.335 span.kind=server system=grpc
time="2022-01-13T08:07:10Z" level=error msg="finished unary call with code Unknown" error="Get \"https://127.0.0.1:62826/version?timeout=32s\": dial tcp 127.0.0.1:62826: connect: connection refused" grpc.code=Unknown grpc.method=Create grpc.service=cluster.ClusterService grpc.start_time="2022-01-13T08:07:10Z" grpc.time_ms=2.614 span.kind=server system=grpc

However API server require authentication, and give this error while unauthenticated connection

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

sumitnagal on Jan 13, 2022