aws-sdk-ruby: Net::HTTP content-length Error When Yielding to Block

Some further info. The exception only happens if I write the chunk to the file. If I write something else, the exception does not occur:

> obj.read { |chunk| f.write("0") }
[AWS S3 200 10.965613 0 retries] get_object(:bucket_name=>"[redacted]",:key=>"[redacted]")
=> {:meta=>{}, :restore_in_progress=>false, :content_type=>"", :etag=>"\"2ba34e5ec00d3c718b104987f18c5069\"", :accept_ranges=>"bytes", :last_modified=>2017-07-18 05:08:05 +0000, :content_length=>28541588, :data=>nil}
> f.rewind
=> 0
> f.read(100)
=> "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"

I added a counter to see how many chunks it processes:

> n = 0
=> 0
> f.truncate(0)
=> 0
> obj.read { |c| n += 1; f.write(c) }
AWS::Core::Http::NetHttpHandler::TruncatedBodyError: content-length does not match
	[... snip ...]
> f.rewind
=> 0
> f.size
=> 34212
> n
=> 8

In a completely un-scientific test of a “handful” of times, it always failed on the 8th chunk and always only wrote 34112 bytes, at least with my specific test file.

I suspected the presence of a raise in ensure block of AWS::Core::Http::NetHttpHandler is masking the original exception. Indeed, after modifying my local copy of the gem to preserve log this exception, I discovered that the origin is an encoding exception (at least in my case): #<Encoding::UndefinedConversionError: "\xCB" from ASCII-8BIT to UTF-8>

A cursory glance at the implementation in the latest version (well, master) implies that this issue is no longer present due to no longer using performing the length check in the context of an ensure clause.

So, this probably is actually fixed. However, for people in older versions of the gem, perhaps ensure your file is open in binary mode first. This worked for me:

> f.truncate(0); obj.read { |c| f.write(c) }
AWS::Core::Http::NetHttpHandler::TruncatedBodyError: content-length does not match
> f.binmode
=> #<File:/var/folders/z1/60kf0jdj67s66tsc1ygv7vcm0000gn/T/x20170718-18872-1ewghxw.ris>
> f.truncate(0); obj.read { |c| f.write(c) }
[AWS S3 200 10.536865 0 retries] get_object(:bucket_name=>"[redacted]",:key=>"[redacted]")
=> {:meta=>{}, :restore_in_progress=>false, :content_type=>"", :etag=>"\"2ba34e5ec00d3c718b104987f18c5069\"", :accept_ranges=>"bytes", :last_modified=>2017-07-18 05:08:05 +0000, :content_length=>28541588, :data=>nil}
> f.size
=> 28541588

Alternatively, set the encoding to the chunk encoding:

> f = Tempfile.new('')
=> #<Tempfile:/var/folders/z1/60kf0jdj67s66tsc1ygv7vcm0000gn/T/20170718-18872-aicy69>
irb(main):092:0> obj.read { |c| f.set_encoding(c.encoding); f.write(c) }
[AWS S3 200 10.536865 0 retries] get_object(:bucket_name=>"[redacted]",:key=>"[redacted]")
=> {:meta=>{}, :restore_in_progress=>false, :content_type=>"", :etag=>"\"2ba34e5ec00d3c718b104987f18c5069\"", :accept_ranges=>"bytes", :last_modified=>2017-07-18 05:08:05 +0000, :content_length=>28541588, :data=>nil}