tailscale: cert: set-dns response: 500 Internal Server Error | Unable to provision new certificate

What is the issue?

I am unable to renew a certificate for a tailscale node, which was previously working. It seems that the Tailscale API throws a HTTP 500 when Tailscale is trying to update/set DNS configurations for the respective ACME DNS Challenge.

It seems to be an issue for this specific node only, I was ably to successfully renew certificates for other nodes.

I did upgrade Tailscale from 1.30.X to latest (1.52.1) in an attempt to resolve this, but this didn’t change anything.

Steps to reproduce

sudo tailscale cert ****.tailnet-****.ts.net

> 500 Internal Server Error: SetDNS "_acme-challenge.****.tailnet-****.ts.net" => "****": set-dns response: 500 Internal Server Error, failed to create DNS record

Additional logs don’t show anything useful:

Nov 13 12:18:31 machine tailscaled[975]: cert("****.tailnet-****.ts.net"): already had ACME account.
Nov 13 12:18:32 machine tailscaled[975]: cert("****.tailnet-****.ts.net"): starting SetDNS call...
Nov 13 12:18:32 machine tailscaled[975]: Accept: TCP{*** > ***} 60 tcp ok
Nov 13 12:18:36 machine tailscaled[975]: cert("****.tailnet-****.ts.net"): getCertPEM: SetDNS "_acme-challenge.****.tailnet-****.ts.net" => ***": set-dns response: 500 Internal Server Error, failed to create DNS record

Are there any recent changes that introduced the issue?

No changes.

OS

Linux

OS version

Ubuntu 20.04.4 LTS

Tailscale version

1.52.1

Other software

No response

Bug report

BUG-221800ba2866e940112ded3ac5100d3c019345ec728be786bc78f1cf3e039be8-20231113064619Z-40e514f598f78137

About this issue

Most upvoted comments

I’m having the same issue on macOS, latest app store version.

2024/09/17 11:39:35 500 Internal Server Error: SetDNS "_acme-challenge.xyzzy.abcd-efg.ts.net" => "qt....84": set-dns response: 500 Internal Server Error, failed to create DNS record
REQ-126598830-d2c3-4516-a925-d1c799a4569e

I’m having this issue currently 500 Internal Server Error: SetDNS "_acme-challenge.name.tail6ceac.ts.net" => "N5ZXGzRb0x0unsAM_yKHQdkTWFY51D8t8ZWDuPQ_akQ": set-dns response: 500 Internal Server Error, failed to create DNS record

I’m experiencing the same issue while configuring the API server proxy in the Kubernetes Operator. The operator pod log contains:

DEBUG tailscaled cert("tailscale-operator.taildxxxx.ts.net"): already had ACME account.
DEBUG tailscaled cert("tailscale-operator.taildxxxx.ts.net"): starting SetDNS call...
DEBUG tailscaled cert("tailscale-operator.taildxxxx.ts.net"): getCertPEM: SetDNS "_acme-challenge.tailscale-operator.taildxxxx.ts.net" => "8Qn...otE": set-dns response: 500 Internal Server Error, failed to create DNS record

Since I’m also using a Tailscale Ingress resource, I checked its pod log as well. It revelaed:

cert("rancher.taildxxxx.ts.net"): getCertPEM: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: rancher.taildxxxx.ts.net, retry after 2024-09-19T04:07:15Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/
http: TLS handshake error from 100.xx.yy.zz:65150: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: rancher.taildxxxx.ts.net, retry after 2024-09-19T04:07:15Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/
cert("rancher.taildxxxx.ts.net"): already had ACME account.

Yes, I’m the one to blame for the too many certificates issue on the Ingress resource. I’m wondering if this also affects the Kubernetes Operator certificate somehow. I already changed the hostname several times to unused values to circumvent this limitation, but the 500 Internal Server Error remains.

I think I’ll just wait for 2024-09-19T04:07:15Z and check if the issue is still reproducible afterwards.