seaweedfs: failed to stream content: read chunk: 500 Internal Server Error
Describe the bug after the update, some files cannot be retrieved all replicas are affected, the client receives 500
direct access like wget http://fast-volume-8.s3-fast-volume.service.dcvcs.consul:8080/1173,010853f6692e8c71?readDeleted=true --spider return 500 Internal Server Error
a small number of files are affected however, it is suspected that the total volume files with problems is up to 30%
filer
filer_server_handlers_read.go:293 failed to stream content /buckets/files/2021/06/18/7110f245-35ae-488b-af28-8a51744e48c6: read chunk: http://fast-volume-8.s3-fast-volume.service.dcvcs.consul:8080/1173,010853f6692e8c71?readDeleted=true: 500 Internal Server Error
filer_server_handlers_read.go:293 failed to stream content /buckets/files/2021/06/18/7110f245-35ae-488b-af28-8a51744e48c6: read chunk: http://fast-volume-12.s3-fast-volume.service.dck12.consul:8080/1173,010853f6692e8c71?readDeleted=true: 500 Internal Server Error
common.go:311 ProcessRangeRequest: read chunk: http://fast-volume-8.s3-fast-volume.service.dcvcs.consul:8080/1173,010853f6692e8c71?readDeleted=true: 500 Internal Server Error
common.go:311 ProcessRangeRequest: read chunk: http://fast-volume-12.s3-fast-volume.service.dck12.consul:8080/1173,010853f6692e8c71?readDeleted=true: 500 Internal Server Error
filechunk_manifest.go:159 read http://fast-volume-8.s3-fast-volume.service.dcvcs.consul:8080/1173,010853f6692e8c71 failed, err: http://fast-volume-8.s3-fast-volume.service.dcvcs.consul:8080/1173,010853f6692e8c71?readDeleted=true: 500 Internal Server Error
filechunk_manifest.go:159 read http://fast-volume-12.s3-fast-volume.service.dck12.consul:8080/1173,010853f6692e8c71 failed, err: http://fast-volume-12.s3-fast-volume.service.dck12.consul:8080/1173,010853f6692e8c71?readDeleted=true: 500 Internal Server Error
volume
common.go:311 ProcessRangeRequest: ReadNeedleData checksum 444161730 expected 1611079626 for Needle: 1173,010853f6692e8c71 Size:2713631, DataSize:2713565, Name:7110f245-35ae-488b-af28-8a51744e48c6, Mime:application/x-gzip Compressed:false
System Setup the problem is 100% present when updating from 3.57 to 3.74 most likely it is when updating from newer versions, but it is impossible to find a sufficient number of confirmations for sure, since there is no tool for check
Expected behavior as I understand, the problem is related to the added new checks at the volume server level if this is the case, then it would be more correct to give the file to the client, but output a message to the logs that there is some problem with the volume it would also be good to have a tool that attempts to download a file directly from volume (and just checks the server response), it will help to find such problems
Additional context at the moment, we have been able to fix this as follows:
- unmount replica volume
- weed fix id after that we get an error like “volumeDataIntegrityChecking failed data file /data/id.dat actual 10485866432 bytes expected 10485866016 bytes” if mount
- weed compact id -method=0 this usually helps, but if the problem persists then weed fix id && weed compact id -method=1
- mount replica
- volume.check.disk -v -nonRepairThreshold 1 -force -slow -syncDeleted -volumeId id at this point, 500 is no longer given to the client, since filer can get at least one copy of the file but sync does not fix this damage, because does not check the ability to return answer
right now it looks like the solution to perform such an action according to all available data and volumes
after performing the weed fix, we received about 20-30% of the volumes that went into RO due to various errors they returned to normal after compact
unfortunately, there is no guarantee that all files will remain in place after that, since it is impossible to quickly check large collections except to download all files
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (17 by maintainers)
@evgmik right. this is because of the old version https://go.dev/play/p/55rBSQguxXP
need to add check for old
crc.Value()for backward compatibility.@chrislusf thank for the hint about crc.Value(), I think this is the key to the problem.
@SmoothDenis , I think I made a fix in PR #6233, if you can please check it
@evgmik I checked, after updating to this version, files became available
@SmoothDenis Of the changes near crc, I only found a transition to a new library in 2021. And it was me 😃
https://github.com/seaweedfs/seaweedfs/commit/865b06b7fe2fe44ffc04cdec0af07aa2a24e7733