cert-manager: Latest ingress-gce does not work with a new ingress resource
/kind bug
cert-manager-v0.3.1
**Ingress isn’t modified and thus he challenge fails when its routed to the proxy behind the ingress **:
The ingress to be modified, thus causing the challenge to be intercepted:
How to reproduce it (as minimally and precisely as possible):
My stack is ingress->service->nginx-proxy
Ingress as follows, which exists and is bound to static global ip - the DNS entry resolves.
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kibana
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "gce"
kubernetes.io/ingress.global-static-ip-name: ${STATIC_IP_NAME}
certmanager.k8s.io/acme-http01-edit-in-place: "true"
certmanager.k8s.io/cluster-issuer: "letsencrypt-issuer"
certmanager.k8s.io/acme-challenge-type: "http01"
labels:
app: kibana
spec:
tls:
- hosts:
- ${DOMAIN_NAME}
secretName: demo-elastic-co
backend:
serviceName: nginx-service
servicePort: 80
rules:
- host: ${DOMAIN_NAME}
Service:
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
namespace: default
spec:
type: NodePort
ports:
- port: 80
name: kibana
selector:
app: nginx
The site is available over http and resolves.
Issuer
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: letsencrypt-issuer
namespace: default
spec:
acme:
# The ACME server URL
server: "https://acme-staging-v02.api.letsencrypt.org/directory"
# Email address used for ACME registration
email: "${EMAIL}"
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-key
# Enable the HTTP-01 challenge provider
http01: {}
Cert
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: demo-elastic-co
namespace: default
spec:
secretName: demo-elastic-co
issuerRef:
name: letsencrypt-issuer
commonName: ${DOMAIN_NAME}
dnsNames:
- ${DOMAIN_NAME}
acme:
config:
- http01:
ingress: kibana
domains:
- ${DOMAIN_NAME}
Creating the issuer and then cert, logs redacted:
0601 17:01:14.034442 1 controller.go:177] certificates controller: syncing item 'default/test-domain-co'
I0601 17:01:14.034567 1 sync.go:239] Preparing certificate default/test-domain-co with issuer
I0601 17:01:14.034582 1 acme.go:159] getting private key (letsencrypt-key->tls.key) for acme issuer default/letsencrypt-issuer
I0601 17:01:14.034925 1 logger.go:27] Calling GetOrder
I0601 17:01:14.187634 1 logger.go:52] Calling GetAuthorization
I0601 17:01:14.259376 1 logger.go:72] Calling HTTP01ChallengeResponse
I0601 17:01:14.259425 1 prepare.go:263] Cleaning up old/expired challenges for Certificate default/test-domain-co
I0601 17:01:14.259447 1 logger.go:47] Calling GetChallenge
I0601 17:01:14.416321 1 helpers.go:162] Found status change for Certificate "test-domain-co" condition "Ready": "False" -> "False"; setting lastTransitionTime to 2018-06-01 17:01:14.416307907 +0000 UTC m=+7713.231874001
I0601 17:01:14.416360 1 sync.go:241] Error preparing issuer for certificate default/test-domain-co: http-01 self check failed for domain "test-domain.example.co"
E0601 17:01:14.423710 1 sync.go:168] [default/test-domain-co] Error getting certificate 'test-domain-co': secret "test-domain-co" not found
E0601 17:01:14.423762 1 controller.go:186] certificates controller: Re-queuing item "default/test-domain-co" due to error processing: http-01 self check failed for domain "test-domain.example.co"
I0601 17:02:14.424648 1 controller.go:177] certificates controller: syncing item 'default/test-domain-co'
I0601 17:02:14.425671 1 sync.go:239] Preparing certificate default/test-domain-co with issuer
I0601 17:02:14.425781 1 acme.go:159] getting private key (letsencrypt-key->tls.key) for acme issuer default/letsencrypt-issuer
I0601 17:02:14.427512 1 logger.go:27] Calling GetOrder
I0601 17:02:14.610652 1 logger.go:52] Calling GetAuthorization
I0601 17:02:14.713524 1 logger.go:72] Calling HTTP01ChallengeResponse
I0601 17:02:14.713564 1 prepare.go:263] Cleaning up old/expired challenges for Certificate default/test-domain-co
I0601 17:02:14.713582 1 logger.go:47] Calling GetChallenge
I0601 17:02:14.817513 1 helpers.go:162] Found status change for Certificate "test-domain-co" condition "Ready": "False" -> "False"; setting lastTransitionTime to 2018-06-01 17:02:14.817499697 +0000 UTC m=+7773.633065800
I0601 17:02:14.817550 1 sync.go:241] Error preparing issuer for certificate default/test-domain-co: http-01 self check failed for domain "test-domain.example.co"
E0601 17:02:14.823613 1 sync.go:168] [default/test-domain-co] Error getting certificate 'test-domain-co': secret "test-domain-co" not found
E0601 17:02:14.823647 1 controller.go:186] certificates controller: Re-queuing item "default/test-domain-co" due to error processing: http-01 self check failed for domain "test-domain.example.co"
In he nginx logs for the app i see the challenge - tweaking the proxy to 404,200 or 301 makes no difference.
35.202.242.207, 35.201.81.158 - - [01/Jun/2018:17:01:14 +0000] "GET /.well-known/acme-challenge/5f_2k1u87-xJ1h4xMjNZN7q9nPlVVSfHVwKH9M58UCw HTTP/1.1" 301 610 "" "Go-http-client/1.1"
35.202.242.207, 35.201.81.158 - - [01/Jun/2018:17:02:14 +0000] "GET /.well-known/acme-challenge/5f_2k1u87-xJ1h4xMjNZN7q9nPlVVSfHVwKH9M58UCw HTTP/1.1" 301 610 "" "Go-http-client/1.1"
Anything else we need to know?:
GCE:
- Kubernetes version (use
kubectl version
):
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-05-12T04:12:12Z", GoVersion:"go1.9.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.2-gke.3", GitCommit:"d2c7a2bd41036f9474287579a725dc54c904e92d", GitTreeState:"clean", BuildDate:"2018-05-23T00:19:39Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration**: GCE on ubuntu
- Install tools: helm
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 36 (8 by maintainers)
Also running into this 😦
I had this same issue, or it at least behaved the same - secret was not being created.
What ended up working for me was the same as mentioned above. I generated a self-signed certificate expiring in one day and manually created the secret.
openssl req -x509 -nodes -days 1 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=subdomain.example.com"
kubectl create secret tls my-secret --key /tmp/tls.key --cert /tmp/tls.crt
After this, cert-manager successfully issued a new certificate.
I am seeing the warning as well, is there any open issue for this that I can subscribe for updates on it? 😄
Awesome, thanks for confirming it has rolled out!
I’m going to close this issue now then as the issue is resolved.
We’ll soon be in a better position to workaround this limitation in future from our end too, to avoid the deprecated behaviour warning.
/close
On Tue, 25 Sep 2018 at 16:56, Brett Curtis notifications@github.com wrote:
This is working now on 1.10.7-gke.2, You’ll see a warming now in your service:
@munnerz
I just encountered this issue using the latest stable/ingress-nginx and stable/cert-manager charts as I write this.
I had to precreate the cert just like for GCE above to make it work with ingress-nginx, other nginx spent all day whining the secret didn’t exist which prevented cert-manager from doing its job.
Did ingress-nginx copy GCE?
The moment I created the secret
ingress-nginx was however generating a self-signed cert, it just seems to completely dropped routing the acme challenge while it was busy throwing a fit over the missing secret
I have opened https://github.com/kubernetes/ingress-gce/pull/388 which will fix this issue.
As another alternative - for now, ingress-gce users can manually specify a Certificate resource. You will need to exclude the TLS section from your Certificate whilst this is provisioning, but once done, you should be good to add it back in referencing the newly created Secret.
same problem here, we are using the manually created tls secret for now.
If i create a cert that is already expired, that should work too yeah?