aws-sdk-ruby: aws-sdk-s3 fails when downloading file of exactly 20971524 bytes from S3 bucket bug

Describe the bug

Hi,

Using FileDownloader’s multithreaded download, downloading a file that is exactly 20971524 bytes long results in an Aws::S3::Errors::InvalidRange (The requested range is not satisfiable) exception.

The issue comes from what looks like a off-by-one error, resulting in that case in an invalid range for that last chunk.

Expected Behavior

File should obviously be downloadable without an exception due to incorrect math/logic 😃

Current Behavior

[Aws::S3::Client 206 12.824679 0 retries] get_object(bucket:"xxx",key:"yyy.jpeg",range:"bytes=0-5242880")  
[Aws::S3::Client 206 12.635049 0 retries] get_object(bucket:"xxx",key:"yyy.jpeg",range:"bytes=5242881-10485761")  
[Aws::S3::Client 206 15.445052 0 retries] get_object(bucket:"xxx",key:"yyy.jpeg",range:"bytes=10485762-15728642")  
[Aws::S3::Client 206 11.991756 0 retries] get_object(bucket:"xxx",key:"yyy.jpeg",range:"bytes=15728643-20971523")  
[Aws::S3::Client 416 0.804207 0 retries] get_object(bucket:"xxx",key:"yyy.jpeg",range:"bytes=20971524-20971524") Aws::S3::Errors::InvalidRange The requested range is not satisfiable

The last chunk is empty/non existant, resulting in the exception.

Reproduction Steps

Any 20971524 bytes long file should show the error, using default download_file options.

Possible Solution

From my understanding, instead of downloading chunks that are MIN_CHUNK_SIZE (= 5242880 bytes) long, the code currently downloads chunks that are MIN_CHUNK_SIZE+1 long.

A file that is 20971524 bytes long should be downloaded as 4 5242880-bytes parts + a last 4-bytes part (20971524 = 5242880 * 4 + 4): 5 parts in total. However, as the code downloads 5242880+1 bytes parts, we currently have 4*5242881 parts plus a final… 0-byte part (which is invalid).

Possible solutions:

• avoid building empty chunks
• fix construct_chunks to build default_chunk_size long parts instead of default_chunk_size+1 (then, also check that a file that is exactly 4*5242880 bytes long gets downloaded in 4 parts w/o a final empty part)

Additional Information/Context

See also https://github.com/aws/aws-sdk-ruby/issues/2561

Gem name (‘aws-sdk’, ‘aws-sdk-resources’ or service gems like ‘aws-sdk-s3’) and its version

aws-sdk-s3

Environment details (Version of Ruby, OS environment)

Ruby 3.1, 3.2, Linux