cert-manager: Failed to perform self check GET request

Describe the bug: When issuing a certificate with ClusterIssuer I get the following output in cert-manager logs:

I0314 05:28:43.956172       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="mydomain" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-8n494" "related_resource_namespace"="testing" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="test-nginx-tls-7lths-4094977200-2514349470" "resource_namespace"="testing" "resource_version"="v1" "type"="HTTP-01" 
I0314 05:28:43.956339       1 ingress.go:98] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="mydomain" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-pzspz" "related_resource_namespace"="testing" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="test-nginx-tls-7lths-4094977200-2514349470" "resource_namespace"="testing" "resource_version"="v1" "type"="HTTP-01" 
E0314 05:28:43.968900       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://mydomain/.well-known/acme-challenge/kVoQ5zQioi_7hK8r8YOcUcrj6G6L2vQ1qCdiHlTtFdI': Get \"https://mydomain:443/.well-known/acme-challenge/kVoQ5zQioi_7hK8r8YOcUcrj6G6L2vQ1qCdiHlTtFdI\": remote error: tls: unrecognized name" "dnsName"="mydomain" "resource_kind"="Challenge" "resource_name"="test-nginx-tls-7lths-4094977200-2514349470" "resource_namespace"="testing" "resource_version"="v1" "type"="HTTP-01"

When I try the URL with curl I get a redirect to the https variant of the URL:

❯ curl -v http://mydomain/.well-known/acme-challenge/kVoQ5zQioi_7hK8r8YOcUcrj6G6L2vQ1qCdiHlTtFdI
*   Trying myip...
* Connected to mydomain (myip) port 80 (#0)
> GET /.well-known/acme-challenge/kVoQ5zQioi_7hK8r8YOcUcrj6G6L2vQ1qCdiHlTtFdI HTTP/1.1
> Host: mydomain
> User-Agent: curl/7.82.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.21.6
< Date: Sun, 13 Mar 2022 21:43:40 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive
< Location: https://mydomain:443/.well-known/acme-challenge/kVoQ5zQioi_7hK8r8YOcUcrj6G6L2vQ1qCdiHlTtFdI
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.21.6</center>
</body>
</html>

And the https variant:

❯ curl -v https://mydomain/.well-known/acme-challenge/kVoQ5zQioi_7hK8r8YOcUcrj6G6L2vQ1qCdiHlTtFdI
*   Trying myip:443...
* Connected to mydomain (myip) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, unrecognized name (624):
* error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name
* Closing connection 0
curl: (35) error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name

Expected behaviour: The certificate is expected to be found on this http(s) URL and get validated

Steps to reproduce the bug:

  1. Install kubernetes on hetzner cloud with kubeone (automatically adds hetzner cloud specific services)
  2. install nginx-ingress-controller with the helm chart
  3. Install cert-manager
  4. Set up cluster issuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: myemail
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx
  1. Set up ingress with tls
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-test-ingress
  namespace: testing
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    # I have tested these as well but they do not help
    #cert-manager.io/issue-temporary-certificate: "true"
    #acme.cert-manager.io/http01-edit-in-place: "true"
spec:
  tls:
  - hosts:
    - "mydomain"
    secretName: test-nginx-tls
  rules:
  - host: mydomain
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-test-service
            port:
              number: 80

Anything else we need to know?:

Environment details::

  • Kubernetes version: 1.23.4
  • Cloud-provider/provisioner: Hetzner Cloud
  • cert-manager version: 1.7.1
  • Install method: static manifests d/l from github to yaml file and then applied
  • The hetzner cloud load balancer that is created for the ingress is not doing the SSL redirect, it is configured to route tcp port 80 to the cluster and tcp port 443 to the cluster

/kind bug

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15

Most upvoted comments

I was able to resolve my issue. I updated nginx to version v2.4.1, and delete the existing ingress. I added acme.cert-manager.io/http01-edit-in-place: "true" to my annotations and then applied my ingress manifest. It didn’t work when I added the edit in place to the exsting Ingress. I observed the HTTP01 solver appended to the Ingress service rules, and removed after the challenge was solved.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: domain-ingress
  namespace: media
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-staging
    acme.cert-manager.io/http01-edit-in-place: "true"
spec:
  tls:
  - hosts: 
      - my.domain.dev
    secretName: domain-tls
  rules:
    - host: my.domain.dev
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-1666294309
                port: 
                  number: 8989
+14
aymond on Oct 30, 2022

I was able to solve this error

"error"="failed to perform self check GET request ...

by adding DNS entries in the cert-manager deployment.

spec.template.spec.dnsConfig

dnsConfig:
  nameservers:
    - 1.1.1.1
    - 8.8.8.8
dnsPolicy: None
+5
rod-stuchi on Jul 19, 2022

Following Kong docs https://docs.konghq.com/kubernetes-ingress-controller/latest/guides/cert-manager/ I ran into this issue, thereby reading https://cert-manager.io/docs/concepts/acme-orders-challenges/ which was pretty helpful in guiding me to diagnose and confirm the problem:

# Excerpt from the output of 'kubectl describe challenge ${challenge-resource-name}' - execute 'kubectl get challenge' first to find the resource name
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://my.domain.dev/.well-known/acme-challenge/3IUHbHR_EQtkyItHMtc7HNMkrnC9gMhhCkPso3FELhw': Get "http://my.domain.dev/.well-known/acme-challenge/3IUHbHR_EQtkyItHMtc7HNMkrnC9gMhhCkPso3FELhw": dial tcp: lookup console.botmast.com on 10.128.0.10:53: no such host

One note for the future though, this:

I was able to solve this error

"error"="failed to perform self check GET request ...

by adding DNS entries in the cert-manager deployment.

spec.template.spec.dnsConfig

dnsConfig:
  nameservers:
    - 1.1.1.1
    - 8.8.8.8
dnsPolicy: None

is inside of this file:

https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yaml  #you should check your current version, this is 1.10.1

It’s a massive file so I would open it up in VS Code and search for kind: Deployment, scroll down a few lines and insert under spec.template.spec as mentioned above.

+1
MarshallBelles on Dec 1, 2022

I found your thread while trying to diagnose my own issue. My failure code is different

"msg"="propagation check failed" "error"="failed to perform self check GET request 'http://domain.tld/.well-known/acme-challenge/BxoPaSKOqSedr3HvG47jQGKNOxFJBTNDTUfoeNnzjEg': Get \"http:/domain.tld/.well-known/acme-challenge/BxoPaSKOqSedr3HvG47jQGKNOxFJBTNDTUfoeNnzjEg\": EOF"

However, I did find this thread which suggests that the http -> https redirection must be disabled,

It makes sense that lets encrypt would not do verification over https because if you need to verify a domain with lets encrypt, you don’t have a certificate yet.

+1
kquinsland on Apr 27, 2022
Read more comments on GitHub
← cert-manager: Failed to install " failed post-install: timed out waiting for the condition"
cert-manager: FailedDiscoveryCheck (403) with cert-manager Webhook →