cert-manager: Latest ingress-gce does not work with a new ingress resource

/kind bug

cert-manager-v0.3.1

**Ingress isn’t modified and thus he challenge fails when its routed to the proxy behind the ingress **:

The ingress to be modified, thus causing the challenge to be intercepted:

How to reproduce it (as minimally and precisely as possible):

My stack is ingress->service->nginx-proxy

Ingress as follows, which exists and is bound to static global ip - the DNS entry resolves.

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.global-static-ip-name: ${STATIC_IP_NAME}
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
    certmanager.k8s.io/cluster-issuer: "letsencrypt-issuer"
    certmanager.k8s.io/acme-challenge-type: "http01"
  labels:
    app: kibana
spec:
  tls:
  - hosts:
    - ${DOMAIN_NAME}
    secretName: demo-elastic-co
  backend:
    serviceName: nginx-service
    servicePort: 80
  rules:
  - host: ${DOMAIN_NAME}

Service:

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: default
spec:
  type: NodePort
  ports:
  - port: 80
    name: kibana
  selector:
    app: nginx

The site is available over http and resolves.

Issuer

---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-issuer
  namespace: default
spec:
  acme:
    # The ACME server URL
    server: "https://acme-staging-v02.api.letsencrypt.org/directory"
    # Email address used for ACME registration
    email: "${EMAIL}"
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-key
    # Enable the HTTP-01 challenge provider
    http01: {}

Cert

---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: demo-elastic-co
  namespace: default
spec:
  secretName: demo-elastic-co
  issuerRef:
    name: letsencrypt-issuer
  commonName: ${DOMAIN_NAME}
  dnsNames:
  - ${DOMAIN_NAME}
  acme:
    config:
    - http01:
        ingress: kibana
      domains:
      - ${DOMAIN_NAME}

Creating the issuer and then cert, logs redacted:

0601 17:01:14.034442       1 controller.go:177] certificates controller: syncing item 'default/test-domain-co'
I0601 17:01:14.034567       1 sync.go:239] Preparing certificate default/test-domain-co with issuer
I0601 17:01:14.034582       1 acme.go:159] getting private key (letsencrypt-key->tls.key) for acme issuer default/letsencrypt-issuer
I0601 17:01:14.034925       1 logger.go:27] Calling GetOrder
I0601 17:01:14.187634       1 logger.go:52] Calling GetAuthorization
I0601 17:01:14.259376       1 logger.go:72] Calling HTTP01ChallengeResponse
I0601 17:01:14.259425       1 prepare.go:263] Cleaning up old/expired challenges for Certificate default/test-domain-co
I0601 17:01:14.259447       1 logger.go:47] Calling GetChallenge
I0601 17:01:14.416321       1 helpers.go:162] Found status change for Certificate "test-domain-co" condition "Ready": "False" -> "False"; setting lastTransitionTime to 2018-06-01 17:01:14.416307907 +0000 UTC m=+7713.231874001
I0601 17:01:14.416360       1 sync.go:241] Error preparing issuer for certificate default/test-domain-co: http-01 self check failed for domain "test-domain.example.co"
E0601 17:01:14.423710       1 sync.go:168] [default/test-domain-co] Error getting certificate 'test-domain-co': secret "test-domain-co" not found
E0601 17:01:14.423762       1 controller.go:186] certificates controller: Re-queuing item "default/test-domain-co" due to error processing: http-01 self check failed for domain "test-domain.example.co"
I0601 17:02:14.424648       1 controller.go:177] certificates controller: syncing item 'default/test-domain-co'
I0601 17:02:14.425671       1 sync.go:239] Preparing certificate default/test-domain-co with issuer
I0601 17:02:14.425781       1 acme.go:159] getting private key (letsencrypt-key->tls.key) for acme issuer default/letsencrypt-issuer
I0601 17:02:14.427512       1 logger.go:27] Calling GetOrder
I0601 17:02:14.610652       1 logger.go:52] Calling GetAuthorization
I0601 17:02:14.713524       1 logger.go:72] Calling HTTP01ChallengeResponse
I0601 17:02:14.713564       1 prepare.go:263] Cleaning up old/expired challenges for Certificate default/test-domain-co
I0601 17:02:14.713582       1 logger.go:47] Calling GetChallenge
I0601 17:02:14.817513       1 helpers.go:162] Found status change for Certificate "test-domain-co" condition "Ready": "False" -> "False"; setting lastTransitionTime to 2018-06-01 17:02:14.817499697 +0000 UTC m=+7773.633065800
I0601 17:02:14.817550       1 sync.go:241] Error preparing issuer for certificate default/test-domain-co: http-01 self check failed for domain "test-domain.example.co"
E0601 17:02:14.823613       1 sync.go:168] [default/test-domain-co] Error getting certificate 'test-domain-co': secret "test-domain-co" not found
E0601 17:02:14.823647       1 controller.go:186] certificates controller: Re-queuing item "default/test-domain-co" due to error processing: http-01 self check failed for domain "test-domain.example.co"

In he nginx logs for the app i see the challenge - tweaking the proxy to 404,200 or 301 makes no difference.

35.202.242.207, 35.201.81.158 - - [01/Jun/2018:17:01:14 +0000] "GET /.well-known/acme-challenge/5f_2k1u87-xJ1h4xMjNZN7q9nPlVVSfHVwKH9M58UCw HTTP/1.1" 301 610 "" "Go-http-client/1.1"
35.202.242.207, 35.201.81.158 - - [01/Jun/2018:17:02:14 +0000] "GET /.well-known/acme-challenge/5f_2k1u87-xJ1h4xMjNZN7q9nPlVVSfHVwKH9M58UCw HTTP/1.1" 301 610 "" "Go-http-client/1.1"

Anything else we need to know?:

GCE:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-05-12T04:12:12Z", GoVersion:"go1.9.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.2-gke.3", GitCommit:"d2c7a2bd41036f9474287579a725dc54c904e92d", GitTreeState:"clean", BuildDate:"2018-05-23T00:19:39Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration**: GCE on ubuntu
  • Install tools: helm

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 36 (8 by maintainers)

Most upvoted comments

Also running into this 😦

I had this same issue, or it at least behaved the same - secret was not being created.

What ended up working for me was the same as mentioned above. I generated a self-signed certificate expiring in one day and manually created the secret.

openssl req -x509 -nodes -days 1 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=subdomain.example.com"

kubectl create secret tls my-secret --key /tmp/tls.key --cert /tmp/tls.crt

After this, cert-manager successfully issued a new certificate.

I am seeing the warning as well, is there any open issue for this that I can subscribe for updates on it? 😄

Awesome, thanks for confirming it has rolled out!

I’m going to close this issue now then as the issue is resolved.

We’ll soon be in a better position to workaround this limitation in future from our end too, to avoid the deprecated behaviour warning.

/close

On Tue, 25 Sep 2018 at 16:56, Brett Curtis notifications@github.com wrote:

This is working now on 1.10.7-gke.2, You’ll see a warming now in your service:

Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP. Note: this behavior is deprecated and will be removed in a future version of ingress-gce

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jetstack/cert-manager/issues/606#issuecomment-424397233, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMbPzFWfxhUNegJqAHp-z6ns5aHHo13ks5uelIwgaJpZM4UXCIT .

This is working now on 1.10.7-gke.2, You’ll see a warming now in your service:

Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP. Note: this behavior is deprecated and will be removed in a future version of ingress-gce

@munnerz

I just encountered this issue using the latest stable/ingress-nginx and stable/cert-manager charts as I write this.

I had to precreate the cert just like for GCE above to make it work with ingress-nginx, other nginx spent all day whining the secret didn’t exist which prevented cert-manager from doing its job.

Did ingress-nginx copy GCE?

W0714 01:16:02.171068       5 controller.go:1020] ssl certificate "dev/web-api-tls-secret" does not exist in local store
10.1.0.4 - [10.1.0.4] - - [14/Jul/2018:01:16:09 +0000] "GET /.well-known/acme-challenge/XXXXXXXXXXXXXXXXXXXXXX-7mmcN-FqwuKc HTTP/1.1" 404 62 "-" "Go-http-client/1.1" 176 0.001 [dev-web-api-service-80] 10.1.0.26:80 31 0.000 404 c0fe1ea8344e36c93c89e6a23cdf9f30
10.1.0.4 - [10.1.0.4] - - [14/Jul/2018:01:16:13 +0000] "GET /.well-known/acme-challenge/XXXXXXXXXXXXXXXXXXXXXX-7mmcN-FqwuKc HTTP/1.1" 404 62 "-" "Go-http-client/1.1" 176 0.001 [dev-web-api-service-80] 10.1.0.25:80 31 0.000 404 f9b367bdfa4df7aa4a2f5f6876f924df
10.1.0.4 - [10.1.0.4] - - [14/Jul/2018:01:16:30 +0000] "GET /.well-known/acme-challenge/XXXXXXXXXXXXXXXXXXXXXX-7mmcN-FqwuKc HTTP/1.1" 404 62 "-" "Go-http-client/1.1" 176 0.001 [dev-web-api-service-80] 10.1.0.7:80 31 0.004 404 51223ce2ad35371671d30a247602259

The moment I created the secret

0714 01:17:56.903217       5 store.go:348] secret dev/web-api-tls-secret was added and it is used in ingress annotations. Parsing...
I0714 01:17:56.904870       5 backend_ssl.go:69] adding secret dev/web-api-tls-secret to the local store
I0714 01:17:57.226319       5 controller.go:177] ingress backend successfully reloaded...
I0714 01:17:58.864434       5 backend_ssl.go:181] updating local copy of ssl certificate dev/web-api-tls-secret with missing intermediate CA certs
I0714 01:18:00.192565       5 controller.go:168] backend reload required
I0714 01:18:00.302296       5 controller.go:177] ingress backend successfully reloaded...
10.1.0.4 - [10.1.0.4] - - [14/Jul/2018:01:18:14 +0000] "GET /.well-known/acme-challenge/XXXXXXXXXXXXXXXXXXXXXX-7mmcN-FqwuKc HTTP/1.1" 200 87 "-" "Go-http-client/1.1" 176 0.026 [dev-cm-acme-http-solver-h49ts-8089] 10.1.0.15:8089 87 0.024 200 489ee18372ea0fe577641c8ace44565b

ingress-nginx was however generating a self-signed cert, it just seems to completely dropped routing the acme challenge while it was busy throwing a fit over the missing secret

I have opened https://github.com/kubernetes/ingress-gce/pull/388 which will fix this issue.

As another alternative - for now, ingress-gce users can manually specify a Certificate resource. You will need to exclude the TLS section from your Certificate whilst this is provisioning, but once done, you should be good to add it back in referencing the newly created Secret.

same problem here, we are using the manually created tls secret for now.

If i create a cert that is already expired, that should work too yeah?