seaweedfs: Slow S3 PUT on Filer

Sponsors SeaweedFS via Patreon https://www.patreon.com/seaweedfs

System Setup SeaweedFS 3.20

• 3 master servers
• 3 filer servers
• 24 volume server. One volume server per HDD.
• MongoDB as filer store

Describe the bug I get significant slowdown on some S3 PUT requests on one of three filers. The delay is exact the same for each PUT request, feels like facing some timeout inside Filer Server.

Though 95% of requests goes well.

Moreover. I’ve mentioned, that slowdown is only on a single bucket. I’ve tried to recreate this bucket in assumption of bad volume distribution. Slowdown came back in some time.

Filer Store has no slowdown, as well as Volume Servers

I’ve scaled filers from 3 to 6 pods. And restarted filer-1 (which has slowdown). Now filer-1 and filer-4 hav both slowdown, on the only same bucket.

Additional context Slow requests are on S3 requests from NodeJS service, used minio-js library, but this lib drops tcp session as soon as PUT request finished with 200 OK answer. Switched to aws-sdk library, which keep tcp connections, but effect is the same.

I tried to tcpdump the requests, looks like the delay is exactly the same, about 10 seconds after last packet of PUT request. Maybe http headers of “slow” PUT requests could be helpful for debugging of the problem.

10:43:31.582810 IP 10.233.99.62.49460 > 10.233.83.42.8333: Flags [.], seq 573:2001, ack 398, win 1412, options [nop,nop,TS val 2093352516 ecr 1883697629], length 1428
PUT /etl-dekart/2022/10/21/10/09eb9cf43c0d49fdb6c997903444f1d6/main-photo.jpg HTTP/1.1
User-Agent: aws-sdk-nodejs/2.1231.0 linux/v16.15.0 callback
Content-Type: image/jpeg
X-Amz-Content-Sha256: 23852ded83e063a88fafbf7cc04819a4f90cae97c4f6640b147e10a1166e1b66
Content-Length: 40632
Host: seaweedfs-s3.seaweedfs:8333
X-Amz-Date: 20221021T074331Z
Authorization: AWS4-HMAC-SHA256 Credential=QYvgrysntQttuO5y/20221021/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3fe3248997012cb3b173505e6089a509c50abc2588f3baca958f7f569f080675
Connection: keep-alive

......JFIF.............C...............
<content>
10:43:31.582889 IP 10.233.99.62.49460 > 10.233.83.42.8333: Flags [.], seq 2001:7713, ack 398, win 1412, options [nop,nop,TS val 2093352516 ecr 1883697629], length 5712
<content>
10:43:31.582947 IP 10.233.99.62.49460 > 10.233.83.42.8333: Flags [.], seq 7713:10569, ack 398, win 1412, options [nop,nop,TS val 2093352516 ecr 1883697629], length 2856
<content>
10:43:31.583006 IP 10.233.83.42.8333 > 10.233.99.62.49460: Flags [.], ack 7713, win 1856, options [nop,nop,TS val 1883697631 ecr 2093352516], length 0
10:43:31.583042 IP 10.233.99.62.49460 > 10.233.83.42.8333: Flags [.], seq 10569:14853, ack 398, win 1412, options [nop,nop,TS val 2093352516 ecr 1883697629], length 4284
<content>
10:43:31.583177 IP 10.233.83.42.8333 > 10.233.99.62.49460: Flags [.], ack 14853, win 1856, options [nop,nop,TS val 1883697632 ecr 2093352516], length 0
10:43:31.583447 IP 10.233.99.62.49460 > 10.233.83.42.8333: Flags [.], seq 14853:16281, ack 398, win 1412, options [nop,nop,TS val 2093352517 ecr 1883697631], length 1428
<content>
10:43:31.583485 IP 10.233.99.62.49460 > 10.233.83.42.8333: Flags [.], seq 16281:20565, ack 398, win 1412, options [nop,nop,TS val 2093352517 ecr 1883697631], length 4284
<content>
10:43:31.583512 IP 10.233.99.62.49460 > 10.233.83.42.8333: Flags [.], seq 20565:24849, ack 398, win 1412, options [nop,nop,TS val 2093352517 ecr 1883697631], length 4284
<content>
10:43:31.583525 IP 10.233.83.42.8333 > 10.233.99.62.49460: Flags [.], ack 20565, win 1856, options [nop,nop,TS val 1883697632 ecr 2093352517], length 0
10:43:31.583551 IP 10.233.83.42.8333 > 10.233.99.62.49460: Flags [.], ack 24849, win 1856, options [nop,nop,TS val 1883697632 ecr 2093352517], length 0
10:43:31.583662 IP 10.233.99.62.49460 > 10.233.83.42.8333: Flags [.], seq 24849:26277, ack 398, win 1412, options [nop,nop,TS val 2093352517 ecr 1883697632], length 1428
<content>
10:43:31.583701 IP 10.233.99.62.49460 > 10.233.83.42.8333: Flags [.], seq 26277:33417, ack 398, win 1412, options [nop,nop,TS val 2093352517 ecr 1883697632], length 7140
<content>
10:43:31.583731 IP 10.233.99.62.49460 > 10.233.83.42.8333: Flags [.], seq 33417:39129, ack 398, win 1412, options [nop,nop,TS val 2093352517 ecr 1883697632], length 5712
<content>
10:43:31.583751 IP 10.233.83.42.8333 > 10.233.99.62.49460: Flags [.], ack 33417, win 1856, options [nop,nop,TS val 1883697632 ecr 2093352517], length 0
10:43:31.583770 IP 10.233.83.42.8333 > 10.233.99.62.49460: Flags [.], ack 39129, win 1856, options [nop,nop,TS val 1883697632 ecr 2093352517], length 0
10:43:41.630353 IP 10.233.83.42.8333 > 10.233.99.62.49460: Flags [P.], seq 398:598, ack 41806, win 1856, options [nop,nop,TS val 1883707679 ecr 2093352518], length 200
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 0
ETag: "e3affc5a9101ee4892848b1996846af9"
Server: SeaweedFS S3
X-Amz-Request-Id: 1666338221630289654
Date: Fri, 21 Oct 2022 07:43:41 GMT