argo-cd: i/o timeout errors in redis and argocd-repo-server PODs

hi I did some testing with delete step by step all polices. The policy which resolves my problem was deleting the argocd-repo-server-network-policy

In my case, there is some wrong in k8s network. Kubernetes network(calico:ipip) doesn’t only use tcp/udp. If you use AWS, check if your security-group allow all-protocol.

In my case the problem was that terraform was overriding the default AWS EKS security groups (allow all), and so the server pod couldn’t communicate with the redis pod. When I added the correct security groups everything started to work as expected.

To help yourself diagnose this problem use “kubectl logs pod/argocd-server…”. Using that I could see that the server pod was timing out when trying to connect to the redis pod, and that helped me narrow it down.

Hi, I am also facing the same problem. Any new workarounds?

Given there were so many different issues I don’t believe the problem is with ArgoCD.

I had been struggling with similar problems this week getting ArgoCD working in a small Vagrant/VirtualBox environment. I switched from Flannel to Calico and everything just magically started working.

Can you confirm the application controller is able to reach the managed cluster’s API server?

Hi, Thanks for the reply. Can you confirm how exactly I do that?

In my case, there is some wrong in k8s network. Kubernetes network(calico:ipip) doesn’t only use tcp/udp. If you use AWS, check if your security-group allow all-protocol.

This got me on the right path, thank you so much!

We’re using the terraform-aws-eks module which only configures security groups for control plane by default. By adding the basic rules as per the complete example, I was able to resolve this issue.

Thank you. Solved by adding the SGs for basic rules as mentioned here.

In my case, there is some wrong in k8s network. Kubernetes network(calico:ipip) doesn’t only use tcp/udp. If you use AWS, check if your security-group allow all-protocol.

This got me on the right path, thank you so much!

We’re using the terraform-aws-eks module which only configures security groups for control plane by default. By adding the basic rules as per the complete example, I was able to resolve this issue.

Hi, i am also facing the same problem, any solution yet ?

I’ve experienced a similar issue, where removing the NetworkPolicy for redis temporarily restored the connectivity. I restored the NetworkPolicy, then restarted the CNI agents on nodes running redis and argocd-server (cilium in my case), and connectivity was restored. I’d be cautious before restarting the cni agents. There was a blip in service communication (as expected). Proceed with caution.

I saw the same issue in v2.0.1 as well, restarting all the pods fixed the issue but not sure what is the cause of it.

Thanks for the hint @jrhoward, but unfortunately this did not solve the issue for me disappointed

It seems that no one of the Argo services can talk with redis.

argocd-server has logs like:

time="2021-06-09T09:42:15Z" level=warning msg="Failed to resync revoked tokens. retrying again in 1 minute: dial tcp 10.43.248.24:6379: i/o timeout"

These are the startup logs of argocd-application-controller:

time="2021-06-09T09:40:06Z" level=info msg="Processing all cluster shards"
time="2021-06-09T09:40:06Z" level=info msg="appResyncPeriod=3m0s"
time="2021-06-09T09:40:06Z" level=info msg="Application Controller (version: v2.0.3+8d2b13d, built: 2021-05-27T17:38:37Z) starting (namespace: argocd)"
time="2021-06-09T09:40:06Z" level=info msg="Starting configmap/secret informers"
time="2021-06-09T09:40:06Z" level=info msg="Configmap/secret informer synced"
time="2021-06-09T09:40:06Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-06-09T09:40:06Z" level=info msg="0xc00097b1a0 subscribed to settings updates"
time="2021-06-09T09:40:06Z" level=info msg="Refreshing app status (normal refresh requested), level (2)" application=drone
time="2021-06-09T09:40:06Z" level=info msg="Starting clusterSecretInformer informers"
time="2021-06-09T09:40:06Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-06-09T09:40:06Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: drone)" application=drone
time="2021-06-09T09:40:06Z" level=info msg="Start syncing cluster" server="https://kubernetes.default.svc"
W0609 09:40:06.719086       1 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0609 09:40:06.815954       1 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
time="2021-06-09T09:40:26Z" level=warning msg="Failed to save clusters info: dial tcp 10.43.248.24:6379: i/o timeout"
time="2021-06-09T09:40:56Z" level=warning msg="Failed to save clusters info: dial tcp 10.43.248.24:6379: i/o timeout"
time="2021-06-09T09:41:16Z" level=warning msg="Failed to save clusters info: dial tcp 10.43.248.24:6379: i/o timeout"
time="2021-06-09T09:41:36Z" level=warning msg="Failed to save clusters info: dial tcp 10.43.248.24:6379: i/o timeout"
time="2021-06-09T09:41:56Z" level=warning msg="Failed to save clusters info: dial tcp 10.43.248.24:6379: i/o timeout"
time="2021-06-09T09:42:16Z" level=warning msg="Failed to save clusters info: dial tcp 10.43.248.24:6379: i/o timeout"
time="2021-06-09T09:42:37Z" level=warning msg="Failed to save clusters info: dial tcp 10.43.248.24:6379: i/o timeout"
...

It seems also that services cannot talk to each other. I found this in argocd-server logs:

time="2021-06-09T08:58:15Z" level=warning msg="finished unary call with code Unavailable" error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 10.43.55.98:8081: connect: connection refused\"" grpc.code=Unavailable grpc.method=GetAppDetails grpc.service=repository.RepositoryService grpc.start_time="2021-06-09T08:58:13Z" grpc.time_ms=2007.533 span.kind=server system=grpc

where 10.43.55.98 is the ClusterIP of the argocd-repo-server service.

I’m very puzzled.

Hi there, had the same issue and stocked in it for several days, logs from the argocd-server showed that the argocd-server connection to the argocd-redis is not established ot got timeout every action it took. so it might be some problems with the argocd-server or its networks. check if the argocd-redis pod it ready,if so, check if the core-dns of the k8s cluster is ready and healthy. In my case it way my calico-node that was not in running mode!

kubectl -n kube-system delete pods calico-node-876rj

and tadaaaa 😃 it was helpful to me. hope works for you.

In my case, I was not specifying a nodeSelector in my Helm values. This caused some of the pods to sometimes land on nodes in a worker group that (similar to what others have described above) did not have the proper security group rules in place.

Using a nodeSelector/affinity to force the pods not to land on these worker node groups solved the issue.

@Tylermarques I resolved the issue. You have to perform the following steps in the solution.

1. add the cluster properly. download Argocd CLI and I use the following command. argocd cluster add kubernetes-admin@kubernetes --in-cluster # The status will be unknow until app deployment
2. Remove following network Policies and restart the pods by deleting them. I. argocd-repo-server-network-policy II. argocd-server-network-policy
3. Please don’t use argo-example repo for the deployment. It won’t work. Instead of it create a public project in your repo. Push the changes. then create ssh public and private keys using this command.

ssh-keygen -t ed25519 -f argocd

Copy Public key in the ssh and GPG keys access section of github as an SSH key. Then go to argocd settings in GUI and add github repo using ssh. There you will need your private key. It will be added successfully. Create the application. When the application will be created, the cluster unknown section will go to successful automatically.

I am facing the following issue.

Unable to create application: application spec for app1 is invalid: InvalidSpecError: repository not accessible: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp: lookup argocd-repo-server: i/o timeout”

I am running my cluster with bare metal Linux VMS. I have deleted all the network policies by reading the above comments. I also tried with different version but still the same issue.

If some has resolved the above issue. could you please share?

By adding the basic rules as per the complete example, I was able to resolve this issue.

This was the solution that worked for me too.

I am still getting the below error after deleting all argocd network policies. I have deployed argocd on minikube cluster, it seems to be ArgoCD does not work well with minikube cluster: Unable to connect SSH repository: connection error: desc = “transport: Error while dialing dial tcp: lookup argocd-repo-server: i/o timeout”

Anyone can help please. it is very annoying

Removing Network Policies helped in my case. I’m using Weave with no other Policies, AgroCD ones were the only NPs there.

Ok on delving deeper into my issue it was actually an SDN issue. I’m running on bare metal. Machines could not reach CoreDNS if they were not on the same machine, if they were on the same machine they couldn’t reach the Redis Server if it was on another machine, so a mixture of DNS lookup failures and network connectivity to Redis

I spoke too soon. The errors are back

Just asking method to verify coz- I could kube exec into app-conroller pod. It chooses non-root user - “argocd” for login in pod. Its debain buster container, where i can’t install ping or ever sudo (to install ping). The plan was to ping kube api server IP (which is K8 master IP) to see if there is communication between two.

argocd@argocd-application-controller-d9d496bdc-hcv7t:~$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"