caddy-ratelimit: I am sure that rate limit not working well

I really understand that you need a way to reproduce the issue to be able to solve it. I don’t know how to reproduce it but I’m 100% sure of the validity of the issue. thanks alot @mholt @mohammed90

I’m not able to reproduce it. I used Vegeta for the load testing, and I don’t see requests passing once the rate limit is hit. Here are the results I witnessed.

Run 1: Without distributed

~ $ cat Caddyfile
{
	debug
	order rate_limit before basicauth
}
localhost {
	log
	rate_limit {
		zone ip_rate {
			key {remote_host}
			events 250
			window 300s
		}
		zone ip_rate_min {
			key {remote_host}
			events 70
			window 100s
		}
	}
	respond "Hi"
}
~ $ echo "GET https://localhost/" | vegeta attack -insecure -duration=40s | tee results.bin | vegeta report
Requests      [total, rate, throughput]         2000, 50.02, 1.75
Duration      [total, attack, wait]             39.981s, 39.98s, 652.577µs
Latencies     [min, mean, 50, 90, 95, 99, max]  496.442µs, 1.237ms, 1.048ms, 1.732ms, 2.116ms, 5.205ms, 21.413ms
Bytes In      [total, mean]                     140, 0.07
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           3.50%
Status Codes  [code:count]                      200:70  429:1930
Error Set:
429 Too Many Requests

Run 2: with distributed

~ $ cat Caddyfile
{
	debug
	order rate_limit before basicauth
}
localhost {
	log
	rate_limit {
		distributed
		zone ip_rate {
			key {remote_host}
			events 250
			window 300s
		}
		zone ip_rate_min {
			key {remote_host}
			events 70
			window 100s
		}
	}
	respond "Hi"
}
~ $ echo "GET https://localhost/" | vegeta attack -insecure -duration=40s | tee results.bin | vegeta report                                         
Requests      [total, rate, throughput]         2000, 50.02, 1.75
Duration      [total, attack, wait]             39.982s, 39.981s, 1.323ms
Latencies     [min, mean, 50, 90, 95, 99, max]  505.503µs, 1.402ms, 988.338µs, 1.979ms, 2.937ms, 8.912ms, 59.632ms
Bytes In      [total, mean]                     140, 0.07
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           3.50%
Status Codes  [code:count]                      200:70  429:1930
Error Set:
429 Too Many Requests

Sample Caddy log line on rate-limit:

ERROR	http.log.access	handled request	{"request": {"remote_ip": "127.0.0.1", "remote_port": "57005", "client_ip": "127.0.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "localhost", "uri": "/", "headers": {"X-Vegeta-Seq": ["1999"], "Accept-Encoding": ["gzip"], "User-Agent": ["Go-http-client/2.0"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "localhost"}}, "bytes_read": 0, "user_id": "", "duration": 0.000067283, "size": 0, "status": 429, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"], "Retry-After": ["0"]}}

If your config has more than what’s shared, please share the full details without redaction. Also check if you’re truly running the configuration you think you’re running.

Unfortunately, I couldn’t reproduce it. But I can see attackers’ IPs passing the web server’s rate limit and reach the application layer rate limit.

I will try to find a way to reproduce it.

Note that any rate limiter using the remote IP as the zone key won’t stop DDoS attacks because you’ll have a lot of different IPs.

Once we have the information needed to reproduce the bug we’ll take a look!