cert-manager: Certificate issuing doesn't work with rewrite-target
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
I tried issuing a certificate for a host. I’m using the nginx ingress controller and its nginx.ingress.kubernetes.io/rewrite-target annotation, which allows rewriting the requested path when forwarding it to a service. The ingress looks something like this (I tried this for my own domain, not example.org, I only use it as an example here):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
name: test
spec:
rules:
- host: example.org
http:
paths:
- backend:
serviceName: test-service
servicePort: 80
path: /test
tls:
- hosts:
- example.org
secretName: example-tls
Logs:
I0127 19:46:14.045231 1 controller.go:187] certificates controller: syncing item 'default/example-tls'
I0127 19:46:14.046410 1 sync.go:107] Error checking existing TLS certificate: secret "example-tls" not found
I0127 19:46:14.046558 1 sync.go:238] Preparing certificate with issuer
I0127 19:46:14.047244 1 prepare.go:239] Compare "" with "https://acme-v01.api.letsencrypt.org/acme/reg/123456789"
I0127 20:01:16.530091 1 helpers.go:165] Setting lastTransitionTime for Certificate "example-tls" condition "Ready" to 2018-01-27 20:01:16.530040382 +0000 UTC m=+515317.420158477
I0127 20:01:16.530692 1 sync.go:242] Error preparing issuer for certificate: error waiting for key to be available for domain "example.org": context deadline exceeded
E0127 20:01:16.552113 1 sync.go:190] [default/example-tls] Error getting certificate 'example-tls': secret "example-tls" not found
E0127 20:01:16.552219 1 controller.go:196] certificates controller: Re-queuing item "default/example-tls" due to error processing: error waiting for key to be available for domain "example.org": context deadline exceeded
I0127 20:01:16.552288 1 controller.go:187] certificates controller: syncing item 'default/example-tls'
...
I think the cause is that cert-manager adds an additional rule to the ingress for the validation:
- backend:
serviceName: cm-main-tls-uwson
servicePort: 8089
path: /.well-known/acme-challenge/XXX
It then tries to parse the challenge from the path (XXX), but it has been rewritten due to the use of the nginx ingress controller annotation.
Maybe it is possible to change the path that is added by cert-manager to just /.well-known/acme-challenge, so even if the rewrite is performed, the challenge component of the path is still avaialable?
What you expected to happen:
A certificate to be issued.
How to reproduce it (as minimally and precisely as possible):
See ingress definition above.
Environment:
-
Kubernetes version (use
kubectl version):Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} -
Cloud provider or hardware configuration**: Self-hosted
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 8
- Comments: 15 (5 by maintainers)
+1 to fail with ingress-shim:
Error checking ACME domain validation: error waiting for key to be available for domain "mycooldomain.com": context deadline exceededHI @mactr0n, right now the better way is to create the Certificate resource yaml yourself, with
ingressClass: nginx, instead of annotating the Ingress and relying oningress-shim.Thanks for the explanation and the example @whereisaaron!
Does cert-manager stop adding it’s own rule to an ingress for the
http01challenge once I create a certificate as you described above, or does it need to exist before creating the ingress which should get a certificate?I worked around it for now, but I think this will still be useful documentation for somebody else who might run into this.