Twitter-API-v2-sample-code: Python and Javascript stream examples break after 5 minutes

Describe the bug Both Python and Javascript example code for streaming break after 5 minutes of retrieving tweets. Javascript just hangs/ends without message, Python throws the following error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 543, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 302, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 598, in read_chunked
    self._update_chunk_length()
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 547, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xxx/.local/lib/python3.6/site-packages/requests/models.py", line 753, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 432, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 626, in read_chunked
    self._original_response.close()
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 320, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "twitter-sample-python.py", line 93, in <module>
    main()
  File "twitter-sample-python.py", line 89, in main
    get_stream(headers, set, bearer_token)
  File "twitter-sample-python.py", line 76, in get_stream
    for response_line in response.iter_lines():
  File "/home/xxx/.local/lib/python3.6/site-packages/requests/models.py", line 797, in iter_lines
    for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
  File "/home/xxx/.local/lib/python3.6/site-packages/requests/models.py", line 756, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

To Reproduce Download example code, set bearer token, run example code. Each time the code is executed, it will stop receiving tweets after the same amount of time, i.e. ~5 minutes. I could reproduce this on two different machines, os (ubuntu+osx), twitter accounts, and three different internet connections.

Expected behavior Stream should not stop.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 8
Comments: 25 (1 by maintainers)

Most upvoted comments

Thanks for all the information here and the workarounds to reconnect. We’ll see what we can do to integrate those changes into future versions of the sample code as needed.

andypiper on Jun 7, 2021

Also experiencing this consistently at 5 minute mark. Noticed after these however, I wouldn’t receive a 429, so wrapped a try/except around the iter_lines() and continued with a while loop.

try:
    for response_line in response.iter_lines():
        if response_line:
            json_response = json.loads(response_line)
            print(json_response)
except requests.exceptions.ChunkedEncodingError:
    encoding_count += 1

A crude patch until Twitter Devs figure out what’s happening on their end?

Da1ne on May 17, 2021

For anyone still following/escaping this error, it would appear the error was on the server side, and was patched yesterday: https://twittercommunity.com/t/filtered-stream-request-breaks-in-5-min-intervals/153926 Good to know we weren’t all going mad!

Da1ne on Jul 14, 2021

Okay I tested this on JavaScript and seems to be fixed.

ShivamJoker on Sep 17, 2021

@andypiper I’ve submitted a pull request for the Python one at #36

Thanks for all the information here and the workarounds to reconnect. We’ll see what we can do to integrate those changes into future versions of the sample code as needed.

pranftw on Jun 7, 2021

I encounter similar issue with “filtered_stream.py” which was working for hours without any problem a month ago.

mustafayd on May 24, 2021