argo-cd: CertManager becomes `SyncError` with ArgoCD v1.1.0-rc1
Describe the bug
I tried to deploy CertManager stable(v0.8.0) with ArgoCD v1.1.0-rc1.
But CertManager sometimes became SyncError
and auto-sync had stopped.
When using Argo CD v1.0.1, this error didn’t occur. I’m afraid v1.1.0-rc1 is degraded…
Detail
In our tryout, CertManager’s CRDs (Certificate
and Issuer
) sometimes become Degraded
, and CertManager’s task was judged as SyncError
.
As far as I searched, the CRDs’ status is False
immediately after the resource is created.
Just at this time, if ArgoCD health check is executed, the task will be judged as SyncError
by following steps.
- The resources are judged as
Degraded
by this Lua script. (This step is unchanged from v1.0.1) - The operation is judged as failed by this code. (Probably spec change in v1.1.0-rc1?)
When doing declarative operations, it sometimes happens that resources are judged as Degraded
.
So I hope auto-sync is not stopped in this situation.
Version v1.1.0-rc1
Logs
When resources are judged as Degraded
time="2019-06-27T05:22:37Z" level=info msg="updating resource result, status: 'Synced' -> 'Synced', phase 'Running' -> 'Failed', message 'issuer.certmanager.k8s.io/cert-manager-webhook-ca created' -> 'Error getting keypair for CA issuer: secret \"cert-manager-webhook-ca\" not found'" application=external-dns kind=Issuer name=cert-manager-webhook-ca namespace=external-dns phase=Sync
time="2019-06-27T05:22:37Z" level=info msg="updating resource result, status: 'Synced' -> 'Synced', phase 'Running' -> 'Failed', message 'certificate.certmanager.k8s.io/cert-manager-webhook-webhook-tls created' -> 'Certificate does not exist'" application=external-dns kind=Certificate name=cert-manager-webhook-webhook-tls namespace=external-dns phase=Sync
When the task becomes SyncError
time="2019-06-27T05:23:39Z" level=info msg=tasks application=external-dns isSelectiveSync=false tasks="[Sync/0 resource /Namespace:external-dns/external-dns obj->obj (Synced,Succeeded,namespace/external-dns configured), Sync/0 resource /ServiceAccount:external-dns/cert-manager obj->obj (Synced,Succeeded,serviceaccount/cert-manager created), Sync/0 resource /ServiceAccount:external-dns/cert-manager-cainjector obj->obj (Synced,Succeeded,serviceaccount/cert-manager-cainjector created), Sync/0 resource /ServiceAccount:external-dns/cert-manager-webhook obj->obj (Synced,Succeeded,serviceaccount/cert-manager-webhook created), Sync/0 resource /ServiceAccount:external-dns/external-dns obj->obj (Synced,Succeeded,serviceaccount/external-dns created), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:external-dns/certificates.certmanager.k8s.io obj->obj (Synced,Succeeded,customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:external-dns/challenges.certmanager.k8s.io obj->obj (Synced,Succeeded,customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:external-dns/clusterissuers.certmanager.k8s.io obj->obj (Synced,Succeeded,customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:external-dns/dnsendpoints.externaldns.k8s.io obj->obj (Synced,Succeeded,customresourcedefinition.apiextensions.k8s.io/dnsendpoints.externaldns.k8s.io created), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:external-dns/issuers.certmanager.k8s.io obj->obj (Synced,Succeeded,customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:external-dns/orders.certmanager.k8s.io obj->obj (Synced,Succeeded,customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created), Sync/0 resource rbac.authorization.k8s.io/ClusterRole:external-dns/cert-manager obj->obj (Synced,Succeeded,clusterrole.rbac.authorization.k8s.io/cert-manager reconciled. clusterrole.rbac.authorization.k8s.io/cert-manager configured), Sync/0 resource rbac.authorization.k8s.io/ClusterRole:external-dns/cert-manager-cainjector obj->obj (Synced,Succeeded,clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector reconciled. clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector configured), Sync/0 resource rbac.authorization.k8s.io/ClusterRole:external-dns/cert-manager-edit obj->obj (Synced,Succeeded,clusterrole.rbac.authorization.k8s.io/cert-manager-edit reconciled. clusterrole.rbac.authorization.k8s.io/cert-manager-edit configured), Sync/0 resource rbac.authorization.k8s.io/ClusterRole:external-dns/cert-manager-view obj->obj (Synced,Succeeded,clusterrole.rbac.authorization.k8s.io/cert-manager-view reconciled. clusterrole.rbac.authorization.k8s.io/cert-manager-view configured), Sync/0 resource rbac.authorization.k8s.io/ClusterRole:external-dns/cert-manager-webhook:webhook-requester obj->obj (Synced,Succeeded,clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester reconciled. clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester configured), Sync/0 resource rbac.authorization.k8s.io/ClusterRole:external-dns/external-dns obj->obj (Synced,Succeeded,clusterrole.rbac.authorization.k8s.io/external-dns created), Sync/0 resource rbac.authorization.k8s.io/ClusterRoleBinding:external-dns/cert-manager obj->obj (Synced,Succeeded,clusterrolebinding.rbac.authorization.k8s.io/cert-manager reconciled. clusterrolebinding.rbac.authorization.k8s.io/cert-manager configured), Sync/0 resource rbac.authorization.k8s.io/ClusterRoleBinding:external-dns/cert-manager-cainjector obj->obj (Synced,Succeeded,clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector reconciled. clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector configured), Sync/0 resource rbac.authorization.k8s.io/ClusterRoleBinding:external-dns/cert-manager-webhook:auth-delegator obj->obj (Synced,Succeeded,clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:auth-delegator created), Sync/0 resource rbac.authorization.k8s.io/ClusterRoleBinding:external-dns/external-dns-viewer obj->obj (Synced,Succeeded,clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created), Sync/0 resource rbac.authorization.k8s.io/RoleBinding:kube-system/cert-manager-webhook:webhook-authentication-reader obj->obj (Synced,Succeeded,rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:webhook-authentication-reader created), Sync/0 resource /Service:external-dns/cert-manager-webhook obj->obj (Synced,Succeeded,service/cert-manager-webhook created), Sync/0 resource /Service:external-dns/external-dns-metrics obj->obj (Synced,Succeeded,service/external-dns-metrics created), Sync/0 resource apps/Deployment:external-dns/cert-manager obj->obj (Synced,Succeeded,deployment.apps/cert-manager created), Sync/0 resource apps/Deployment:external-dns/cert-manager-cainjector obj->obj (Synced,Succeeded,deployment.apps/cert-manager-cainjector created), Sync/0 resource apps/Deployment:external-dns/cert-manager-webhook obj->obj (Synced,Running,deployment.apps/cert-manager-webhook created), Sync/0 resource apps/Deployment:external-dns/external-dns obj->obj (Synced,Succeeded,deployment.apps/external-dns created), Sync/0 resource apiregistration.k8s.io/APIService:external-dns/v1beta1.admission.certmanager.k8s.io obj->obj (Synced,Running,apiservice.apiregistration.k8s.io/v1beta1.admission.certmanager.k8s.io created), Sync/0 resource admissionregistration.k8s.io/ValidatingWebhookConfiguration:external-dns/cert-manager-webhook obj->obj (Synced,Succeeded,validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created), Sync/0 resource certmanager.k8s.io/Certificate:external-dns/cert-manager-webhook-ca obj->obj (Synced,Succeeded,Certificate is up to date and has not expired), Sync/0 resource certmanager.k8s.io/Issuer:external-dns/cert-manager-webhook-ca obj->obj (Synced,Failed,Error getting keypair for CA issuer: secret \"cert-manager-webhook-ca\" not found), Sync/0 resource certmanager.k8s.io/Issuer:external-dns/cert-manager-webhook-selfsign obj->obj (Synced,Succeeded,issuer.certmanager.k8s.io/cert-manager-webhook-selfsign created), Sync/0 resource certmanager.k8s.io/Certificate:external-dns/cert-manager-webhook-webhook-tls obj->obj (Synced,Failed,Certificate does not exist), Sync/1 resource certmanager.k8s.io/ClusterIssuer:external-dns/clouddns nil->obj (,,)]"
time="2019-06-27T05:23:39Z" level=info msg="updating resource result, status: 'Synced' -> 'Synced', phase 'Running' -> 'Succeeded', message 'deployment.apps/cert-manager-webhook created' -> 'deployment.apps/cert-manager-webhook created'" application=external-dns kind=Deployment name=cert-manager-webhook namespace=external-dns phase=Sync
time="2019-06-27T05:23:39Z" level=info msg="updating resource result, status: 'Synced' -> 'Synced', phase 'Running' -> 'Succeeded', message 'apiservice.apiregistration.k8s.io/v1beta1.admission.certmanager.k8s.io created' -> 'all checks passed'" application=external-dns kind=APIService name=v1beta1.admission.certmanager.k8s.io namespace=external-dns phase=Sync
time="2019-06-27T05:23:39Z" level=info msg="Updating operation state. phase: Running -> Failed, message: 'one or more tasks are running' -> 'one or more synchronization tasks completed unsuccessfully'" application=external-dns
time="2019-06-27T05:23:39Z" level=info msg="sync/terminate complete" application=external-dns
time="2019-06-27T05:23:39Z" level=info msg="Sync operation to 62802b64bf4a3df19bce40e4f44354a39655b5b1 failed: one or more synchronization tasks completed unsuccessfully" application=external-dns reason=OperationCompleted type=Warning
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 17 (16 by maintainers)
Commits related to this issue
- Does not fail a sync if there is only one wave and zero hooks. Closes #1826 — committed to alexec/argo-cd by alexec 5 years ago
Fixed in https://github.com/argoproj/argo-cd/pull/1921
Nice. We should make the health check for APIService a built in (native golang) one so everyone will benefit from this.