cert-manager: Cloudflare DNS resolver fails: Error: 6003: Invalid request headers

For what it’s worth I was able to fix my issue by changing from the documented Secret. Changes made:

• Added to the cert-manager namespace
• Used data as opposed to stringData because I base64 encoded the secret

I’m using API Token from Cloudflare with a ClusterIssuer. Didn’t need to adjust from the recommended minimal permission requirements in the docs.

export API_TOKEN=$(echo -n '-myVeryS3cretAPITokenfromCloudfl4re' | base64)
cat <<EOF | kubectl apply -f - 
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
data:
  api-token: ${API_TOKEN}
EOF

Before those changes I was seeing the below errors:

• Secret not found (fixed by changing namespace)
• Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header (fixed with data vs stringData)

I was just able to deploy it using an api token generated from cloudflare.

Follow the steps here : https://blog.darkedges.com/2020/05/04/cert-manager-kubernetes-cloudflare-dns-update/

but, instead of using apiKeySecretRef, using a token here, use apiTokenSecretRef.

You do not need to base64 encode the secret. I read somewhere else that they only got it to work by doing that, so I was on the wrong track.

apiTokenSecretRef

For what it’s worth I was able to fix my issue by changing from the documented Secret. Changes made:

• Added to the cert-manager namespace
• Used data as opposed to stringData because I base64 encoded the secret

I’m using API Token from Cloudflare with a ClusterIssuer. Didn’t need to adjust from the recommended minimal permission requirements in the docs.

export API_TOKEN=$(echo -n '-myVeryS3cretAPITokenfromCloudfl4re' | base64)
cat <<EOF | kubectl apply -f - 
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
data:
  api-token: ${API_TOKEN}
EOF

Before those changes I was seeing the below errors:

• Secret not found (fixed by changing namespace)
• Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header (fixed with data vs stringData)

Confirm changing stringData -> data in the Secret (which needs to be in cert-manager namespace) worked. Sigh the hours we burn…

I did fixed with global token using

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-key
  namespace: cert-manager
type: Opaque
stringData:
  api-token: xxxxxxxxxx
----
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: test@test.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        cloudflare:
          email: test@test.com
          apiKeySecretRef:
            name: cloudflare-api-key
            key: api-token

siggghhh, I tried different reconfigurations etc. but as it turns out the 5th new generated Cloudflare API Token somehow worked. I don’t think it really made a difference that I started double quoting the String in the secret. Maybe someone else encountering this issue can confirm, that a new, correct API Token resolves this issue. Cert-Manager documented which access roles are required: https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/#api-tokens. For my part, idk what the problem with my first attempts was…

This got me thinking as I just had this problem on a new cluster and api key. This is happening when the cloudflare api key has a - (hyphen) in the key. My first and second key I had tried both had hyphens in them. I redeployed certmanager with the old keys to make sure it was still broken before I tried the fix and it was. I rotated the key until I had one without a hyphen and redeployed. It then worked as expected. I’d consider this a bug that needs to be fixed.

I’ve just been setting this up myself and was curious why my apiTokenSecretRef was not working. Turns out it does work if I don’t base64 encode the secret. This seems odd and I don’t know enough to dig through and find out why this is the case.

Ignore me, this was my lack of understanding of the difference between data and stringData!

@luishdez which version of certmanager are you running?

I’m on the latest version of certmanager, your yaml simply results in Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header