cert-manager: ingress-gce: ACME certificates fail to issue for the first time
Is your feature request related to a problem? Please describe.
When using ingress-shim to automatically generate a Certificate resource for a GCE ingress using v0.6.0, if an existing Secret containing a signed keypair does not exist ahead of time, ingress-gce will enter a state where it won’t update the GCLB in the google api to add the HTTP01 challenge solver paths.
Users will see an error such as:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ADD 22m loadbalancer-controller cdn-how-to/my-ingress
Normal CreateCertificate 22m cert-manager Successfully created Certificate "test1.rimusz.xyz"
Warning Sync 14m loadbalancer-controller Error during sync: Error running backend syncing routine: unable to find nodeport {cdn-how-to/cm-acme-http-solver-dqvf5/8089 32735 8089 HTTP 8089 false <nil>} in any service
Normal CREATE 14m loadbalancer-controller ip: 35.201.110.7
Warning Sync 33s (x33 over 21m) loadbalancer-controller Error during sync: Error running load balancer syncing routine: Cert creation failures - k8s-ssl-0fd30a7e225361e2-e3b0c44298fc1c14--1cea66e2afee452f Error:googleapi: Error 400: Invalid value for field 'resource.certificate': ''. A certificate must be specified for SSL certificate creation., invalid
when running kubectl describe
on their ingress resource.
Describe the solution you’d like
This is a new bug introduced in v0.6 - notably because as part of v0.6, we now generate a private key for the certificate before obtaining the signed certificate. This means we leave the Secret resource with only a tls.key
entry, and no tls.crt
.
This leads ingress-gce to throw errors, because tls.crt
is empty. It then refuses to update the paths, which prevents the HTTP01 challenge passing.
We should automatically generate a self signed certificate if only a private key has been generated and no certificate already exists. This is probably useful behaviour for all Issuer types that only return private keys that need to be persisted sometimes.
Describe alternatives you’ve considered
In the meantime, users can either:
- create a secret resource with some kind of dummy secret in ahead of time
- remove the
ingress.spec.tls[]
entry temporarily, and instead manually create the Certificate resource - ingress-gce will not try and enable TLS on the LB until you manually add this entry again. This will allow the Certificate to be issued. Upon renewal, there will already be an existing secret there, meaning this problem won’t be hit again at renew time.
Additional context
This is a regression from v0.5, and whilst it doesn’t break existing deployments, it does cause problems for new users deploying v0.6 for the first time.
Environment details (if applicable):
- Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): GKE
- cert-manager version (e.g. v0.4.0): v0.6.x
/kind bug /priority important-soon /milestone v0.7 /area acme /area acme/http01 cc @rimusz @ahmetb
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 3
- Comments: 17 (2 by maintainers)
I’ve got just one domain in my ingress hosts list and I know it’s pointing to the correct IP. I’m still getting these events.
Error during sync: error running load balancer syncing routine: loadbalancer example-nginx--0a84289edb80f53e does not exist: Cert creation failures - k8s-ssl-fe4dc2b03cd5fcb9-e3b0c44298fc1c14--0a84289edb80f53e Error:googleapi: Error 400: Invalid value for field 'resource.certificate': ''. A certificate must be specified for SSL certificate creation., invalid
If I’m reading that correctly, it needs a certificate to be able to create a certificate. Is that correct? I’m on v0.11 with ingress-nginx.
Hi All, I’m getting following error ok GKE and my domain is hosted on godaddy.com
What I’m missing here /Shiv
I had same issue and generated a self-signed cert using the
tls.key
to sign and place them in the secrets and errors ended. I found out, however, that the issue wasn’t the signing but instead thehosts
list was the issue and I had a couple domains that were not pointing to the static IP so theorder
with cert manager never completed. Once I removed those invalid URLs from hosts, and re-applied the ingress manifest, it worked fine.Hope that helps and tl;dr is make sure every domain in ingress file resolves to the IP served by ingress (I use external DNS with GoDaddy)
I had the same problem with ingress-gce on GKE platform, solved by:
After some time ingress will pull new certificate from lets encrypt and start using it. Note: I’m not using Certificate kind.