requests: Error decoding chunked response in _update_chunk_length: ValueError: invalid literal for int() with base 16: b'HTTP/1.1 200 OK\r\n'

Decoding a response with chunked encoding fails because it seems to be reading from the HTTP status line instead of the start of the body.

Expected Result

Successfully return a 200 response object.

Actual Result

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 535, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b'HTTP/1.1 200 OK\r\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 298, in _error_catcher
    yield
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 590, in read_chunked
    self._update_chunk_length()
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 539, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(17 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 719, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 428, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 618, in read_chunked
    self._original_response.close()
  File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 316, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(17 bytes read)', IncompleteRead(17 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 641, in send
    r.content
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 797, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 722, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(17 bytes read)', IncompleteRead(17 bytes read))

Reproduction Steps

import requests
requests.get('https://my.virginmoney.com.au/AUVMA/JSO/signon/DisplayUsernameSignon.do')

System Information

requests 2.13.0 urllib3 1.20 Python 3.6.2 Fedora 26

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 6
  • Comments: 25 (5 by maintainers)

Commits related to this issue

Most upvoted comments

i’m facing same kind of issue. any idea how to bypass this?

<Response [200]>
Traceback (most recent call last):
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 539, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 302, in _error_catcher
    yield
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 594, in read_chunked
    self._update_chunk_length()
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 543, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/models.py", line 738, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 432, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 622, in read_chunked
    self._original_response.close()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 100, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 320, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "loginscheck.py", line 49, in <module>
    f.write(req.text)
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/models.py", line 840, in text
    if not self.content:
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/models.py", line 816, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/models.py", line 741, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

I’m running into this exact issue as well, with an identical traceback as https://github.com/requests/requests/issues/4248#issuecomment-398007524. I’ve tried reverting requests as far back as 2.9.1, and as far forward as the current release (2.20.1), with a few different combinations of urllib3 versions in each case - they all give the exact same traceback.

Is there a release which we can use until this is fixed, or do we have to patch our requests with the above corrected def?

I solved this problem by adding one line code.

python3 on Mac urllib3 == 1.23 requests == 2.19.1

Traceback (most recent call last): File “/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/response.py”, line 574, in _update_chunk_length self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b’’

so, I open the file “…/urllib3/response.py”, add line = (len(line)>0 and line or "0") in 571, like below:

    def _update_chunk_length(self):
        # First, we'll figure out length of a chunk and then
        # we'll try to read it from socket.
        if self.chunk_left is not None:
            return
        line = self._fp.fp.readline()
        line = line.split(b';', 1)[0]
        line = (len(line)>0 and line or "0")     # added this line
        try:
            self.chunk_left = int(line, 16)
        except ValueError:
            # Invalid chunked protocol response, abort.
            self.close()
            raise httplib.IncompleteRead(line)

This works fine with requests 2.10.0 so something regressed between then and 2.13.0. Also tested with 2.18.4 and the bug exists as above.

We temporarily repaired it in this way. I wonder if there will be any problem.

def _update_chunk_length(self):
    # First, we'll figure out length of a chunk and then
    # we'll try to read it from socket.
    if self.chunk_left is not None:
        return
    line = self._fp.fp.readline()
    line = line.split(b';', 1)[0]
    try:
        if len(line) == 0:
            self.chunk_left = 0
        else:
            self.chunk_left = int(line, 16)
    except ValueError:
        # Invalid chunked protocol response, abort.
        self.close()
        raise httplib.IncompleteRead(line)

@KonMann you seem to have more time than me. Go for it.