certspotter: 429 Too Many Requests from TrustAsia and DigiCert logs

Since Monday around 16:00 CET, my Cert Spotter installation has been outputting this on every run:

2021/11/08 16:00:04 https://ct.trustasia.com/log2022/: Error retrieving STH from log: GET https://ct.trustasia.com/log2022/ct/v1/get-sth: 429 Too Many Requests (<html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>openresty</center>
</body>
</html>
)
2021/11/08 16:00:04 https://ct.trustasia.com/log2023/: Error retrieving STH from log: GET https://ct.trustasia.com/log2023/ct/v1/get-sth: 429 Too Many Requests (<html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>openresty</center>
</body>
</html>
)

I know that Cert Spotter downloads a list of CT logs to check from certspotter.org before every run, so I guess that something in that list was recently changed. Is this expected behaviour and is TrustAsia’s WAF a bit trigger happy or is Cert Spotter incorrectly sending lots of meaningless requests for the empty 2022/2023 TrustAsia logs?

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 24 (10 by maintainers)

Commits related to this issue

certspotter: switch to a local CT logs list certspotter by default fetches the list of CT logs to monitor from https://loglist.certspotter.org/monitor.json. It does however support reading this log f... — committed to wikimedia/operations-puppet by azadi 2 years ago

Most upvoted comments

A fix is forthcoming.

AGWA on May 31, 2022

@imrejonk I also observed 429 errors from Yeti 2022-2 at around the same timeframe. I’m drafting an email to ct-policy that will reference this as well as some other observations about this log.

AGWA on Jul 22, 2022

We’re six months later now. I encountered a few (<10) TrustAsia rate limit errors in the meantime. Since yesterday however I’m seeing many 429 Too Many Requests messages from https://yeti2022-2.ct.digicert.com/log/ (times are in CEST):

2022/05/01 07:17:18 https://yeti2022-2.ct.digicert.com/log/: Problem fetching entries 479921031 to 479921262 from log: GET https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=479921031&end=479921262: 429 Too Many Requests (<html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>nginx</center>
</body>
</html>
)
2022/05/01 07:17:18 https://yeti2022-2.ct.digicert.com/log/: Error scanning log (if this error persists, it should be construed as misbehavior by the log): GET https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=479921031&end=479921262: 429 Too Many Requests (<html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>nginx</center>
</body>
</html>
)

We run Cert Spotter once every hour and this message was logged almost every hour.

I’d like to turn this bug report into a feature request. It would be great if the number of requests that Cert Spotter does per second can be limited so that excessive requests can be prevented.

imrejonk on May 2, 2022

Thanks for the swift reply, I’ve contacted the TrustAsia log operators and will update this issue once I get a response.

imrejonk on Nov 10, 2021