urllib3: Getting exception in _update_chunk_length
Hi, While requesting a particular URL, I came across this error
File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 601, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 360, in _error_catcher
yield
File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 666, in read_chunked
self._update_chunk_length()
File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 605, in _update_chunk_length
raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)
During handling of the above exception, another exception occurred:
aceback (most recent call last):
File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/models.py", line 750, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 490, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 694, in read_chunked
self._original_response.close()
File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 378, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "parse_dumped_data.py", line 87, in <module>
Parse(f, entity)
File "parse_dumped_data.py", line 17, in __init__
self.parse_records()
File "parse_dumped_data.py", line 32, in parse_records
data_fields = self.get_data(record.get('data'))
File "parse_dumped_data.py", line 50, in get_data
data['image_url'] = self.get_image_url(data.get('image'), _id)
File "parse_dumped_data.py", line 64, in get_image_url
resp = requests.get(url)
File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/sessions.py", line 686, in send
r.content
File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/models.py", line 828, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/models.py", line 753, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
This was reported in requests module here
I fixed it as -
def _update_chunk_length(self):
# First, we'll figure out length of a chunk and then
# we'll try to read it from socket.
if self.chunk_left is not None:
return
line = self._fp.fp.readline()
line = line.split(b';', 1)[0]
try:
if len(line) == 0:
self.chunk_left = 0
else:
self.chunk_left = int(line, 16)
except ValueError:
# Invalid chunked protocol response, abort.
self.close()
raise httplib.IncompleteRead(line)
or a one liner as
def _update_chunk_length(self):
# First, we'll figure out length of a chunk and then
# we'll try to read it from socket.
if self.chunk_left is not None:
return
line = self._fp.fp.readline()
line = line.split(b';', 1)[0]
line = (len(line)>0 and line or "0") # added this line
try:
self.chunk_left = int(line, 16)
except ValueError:
# Invalid chunked protocol response, abort.
self.close()
raise httplib.IncompleteRead(line)
Is it worth giving a PR regarding this??
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 20 (4 by maintainers)
Ok, I confirmed this is a bug in openssl 1.1 versions, working in openssl 1.0 Will fill the bug there, sorry for the noise !
@mbatle I guess I should be watching https://github.com/openssl/openssl/issues/9360 ?
I was thinking about this as well. In the original post, we clearly have a response we could attach. Perhaps
read_chunkedcould includeselfin the exception to make it easier?I stepped through urllib3 during the read. The server was returning a 500. The response headers said it was chunked but it wasn’t – I believe the body was empty.
So in my case this wasn’t a bug in urllib3; the server definitely sent a spec-violating response. I’m just going to leave this here to remind other folks that an “incomplete read” may be because the data was never written.