cert-manager: Operation cannot be fulfilled on certificates.cert-manager.io the object has been modified
Hello,
I’m trying to issue certificate from a k8s Ingress, the cert is not becomes ready, the logs from cert-manager pod print this:
E1208 16:01:50.827738 1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"example-com-test-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="example/example-com-test-tls"
E1208 16:01:50.872116 1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"example-com-test-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="example/example-com-test-tls"
Here is the ingress part of my value.yaml :
ingress:
enabled: true
labels:
traffic: "external"
use-dns01-solver: "true"
annotations: |
kubernetes.io/ingress.class: haproxy
cert-manager.io/cluster-issuer: letsencrypt-prod
ingress.kubernetes.io/whitelist-source-range: "0.0.0.0"
kubernetes.io/ingress.allow-http: "false"
ingress.kubernetes.io/ssl-passthrough: "true"
hosts:
- host: example-test.hubstairs.com
paths:
- /
serviceName: myapp
servicePort: 3031
tls:
- secretName: example-com-test-tls
hosts:
- 'example-test.hubstairs.com'
The certificate has been successfully issue without error :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 32s cert-manager Issuing certificate as Secret does not exist
Normal Generated 32s cert-manager Stored new private key in temporary Secret resource "example-com-test-tls-6bt7d"
Normal Requested 32s cert-manager Created new CertificateRequest resource "example-com-test-tls-xk85p"
Normal Issuing 12s cert-manager The certificate has been successfully issued
When I try to access to url in a browser, I have an SSL error.
Environment details::
- Kubernetes version: 1.17
- Cloud-provider/provisioner: Google cloud
- cert-manager version: 1.1.0
- Install method: helm upgrade --install jetstack/cert-manager --version 1.1.0 -f value.yaml --debug
/kind bug
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 81
- Comments: 40 (4 by maintainers)
Okay, my bad, I have messed up the annotation:
cert-manager.io/issuer: letsencrypt-stage->cert-manager.io/cluster-issuer: letsencrypt-stageEverything seems to work now, sorry for the needless spam.
As @irbekrm detailed, the message:
is benign 99.9% of the time. This message does not actually relate to the issue you may observe; Unfortunately, the message is unclear and people may think it is a symptom of their issue.
In 1.4, we changed the message a bit to clarify that. The level was changed from “error” to “info” and the message was reworded:
It seems like the message still looks like something “failed”. I will look into ways of making this logline more understandable; either moving it to
-v=4(since this message is useful while developing to discover when two controllers are fighting on the same resource).⚠ To anyone reading this thread: please open a new issue with the detailed logs that led to your issue, and keep in mind that the above message is very most likely not the culprit.
Same here, I had a failed CM installation, which had to be reinstalled. After that, some of the certificates had the
the object has been modified; please apply your changes to the latest version and try againerror.Just a quick workaround solution I used:
Some of it might not be necessary, but with letsencrypt it’s a non-destructive removal as everything will be recreated. Hope it helps.
@meyskens
I did an upgrade to the latest cert-manager and all seems to be working, it’s just that we notice this error with every (each and everyone) certificate request and renewal.
How should I look at this error? If cert-manager will work around these itself, should I consider it very verbose info output that causes some panic on my end or is this because of a bug or misconfiguration that is triggered?
If it’s just very verbose info output about an issue that cert-manager will work around itself I’d love to be able to suppress it as it causes panic without reason. If it’s not just extremely verbose info I’d love to have a fix so it doesn’t appear anymore in my logs.
Edit: Forgot to ask my question: Can i fix the appearance of this “error” with configuration or is it something that should be fixed in the project itself?
@fliphess cert-manager will work around these itself
Somehow my clusterissuers got deleted. Restoring them fixed this for me.
Hi folks,
This line in logs does not indicate an issue in itself, see the discussion around #3794 . The wording was changed slightly as it seemed to be misleading when folks are debugging other issues.
There could be other issues why the certs are not getting renewed- it might make sense to open another issue with some more detail (the
Certificate,CertificateRequestand some more logs from cert-manager pod (you can increase the verbosity level with--v=5on controller)). After briefly looking at the issues reported above, I think they likely have different causes.Using K8s 1.18 on AWS, cert-manager 1.4.0, clusterIssuer: letsencrypt-prod. ( Deleted old cert-manager. Installed 1.4.0 and reconf’d clusterIssuer.)
I deleted the cert which seemed to be causing the issue, and it re-issued, but the cert-manager pod is still kicking the error in logs.
I’m stuck at this problem.
In my case, every deployment is a completely fresh environment, but no matter what I try, I get the “the object has been modified; please apply your changes to the latest version and try again” line.
Tried:
Install method: via Rancher 2.5.7 using
jetstack/cert-managerHelm3 catalogue from https://charts.jetstack.ioIssuer:
Ingress:
Had same problem, using cert-manager v1.0.0 on google gke. 1 certificate was expired, removed the Certificate , and recreated it. manager fetched a new certificate after this action and service was restored.
@meyskens : for my understanding, this is still a open issue which will be fixed or is it already fixed on 1.x version ?
@meyskens I going to update the HAProxy controller through the latest version. I’ll keep you informed if the problem goes away.
I see the same errors in the log but everything works fine. You seem to have omitten the non error lines, which show that it eventually worked.
I would guess that this error log lines are not related to your actual problem. I opened a separate issue because I am not sure if we are dealing with the same problem. https://github.com/jetstack/cert-manager/issues/3516