urllib3: Getting exception in _update_chunk_length

Hi, While requesting a particular URL, I came across this error

File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 601, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 360, in _error_catcher
    yield
  File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 666, in read_chunked
    self._update_chunk_length()
  File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 605, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

  aceback (most recent call last):
  File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 490, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 694, in read_chunked
    self._original_response.close()
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/root/shivam/python3_env/lib/python3.5/site-packages/urllib3/response.py", line 378, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "parse_dumped_data.py", line 87, in <module>
    Parse(f, entity)
  File "parse_dumped_data.py", line 17, in __init__
    self.parse_records()
  File "parse_dumped_data.py", line 32, in parse_records
    data_fields = self.get_data(record.get('data'))
  File "parse_dumped_data.py", line 50, in get_data
    data['image_url'] = self.get_image_url(data.get('image'), _id)
  File "parse_dumped_data.py", line 64, in get_image_url
    resp = requests.get(url)
  File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/sessions.py", line 686, in send
    r.content
  File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/models.py", line 828, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/root/shivam/python3_env/lib/python3.5/site-packages/requests/models.py", line 753, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

This was reported in requests module here

I fixed it as -

def _update_chunk_length(self):
    # First, we'll figure out length of a chunk and then
    # we'll try to read it from socket.
    if self.chunk_left is not None:
        return
    line = self._fp.fp.readline()
    line = line.split(b';', 1)[0]
    try:
        if len(line) == 0:
            self.chunk_left = 0
        else:
            self.chunk_left = int(line, 16)
    except ValueError:
        # Invalid chunked protocol response, abort.
        self.close()
        raise httplib.IncompleteRead(line)

or a one liner as

    def _update_chunk_length(self):
        # First, we'll figure out length of a chunk and then
        # we'll try to read it from socket.
        if self.chunk_left is not None:
            return
        line = self._fp.fp.readline()
        line = line.split(b';', 1)[0]
        line = (len(line)>0 and line or "0")     # added this line
        try:
            self.chunk_left = int(line, 16)
        except ValueError:
            # Invalid chunked protocol response, abort.
            self.close()
            raise httplib.IncompleteRead(line)

Is it worth giving a PR regarding this??

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 20 (4 by maintainers)

Most upvoted comments

Ok, I confirmed this is a bug in openssl 1.1 versions, working in openssl 1.0 Will fill the bug there, sorry for the noise !

I’d suggest that the wrapper exception contain the HTTP response code since that might help diagnose a broken server.

I was thinking about this as well. In the original post, we clearly have a response we could attach. Perhaps read_chunked could include self in the exception to make it easier?

I stepped through urllib3 during the read. The server was returning a 500. The response headers said it was chunked but it wasn’t – I believe the body was empty.

So in my case this wasn’t a bug in urllib3; the server definitely sent a spec-violating response. I’m just going to leave this here to remind other folks that an “incomplete read” may be because the data was never written.