twitter-scraper: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Running the minimal example crashes with an error:
$ ipython
Python 3.8.3 (default, May 19 2020, 13:54:14)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.17.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from twitter_scraper import get_tweets
In [2]: for tweet in get_tweets('twitter', pages=1):
...: print(tweet['text'])
...:
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
<ipython-input-2-bd0d769d18f8> in <module>
----> 1 for tweet in get_tweets('twitter', pages=1):
2 print(tweet['text'])
3
~/opt/miniconda3/lib/python3.8/site-packages/twitter_scraper-0.4.3-py3.8.egg/twitter_scraper/modules/tweets.py in get_tweets(query, pages)
169 pages += -1
170
--> 171 yield from gen_tweets(pages)
172
173
~/opt/miniconda3/lib/python3.8/site-packages/twitter_scraper-0.4.3-py3.8.egg/twitter_scraper/modules/tweets.py in gen_tweets(pages)
34 while pages > 0:
35 try:
---> 36 json_response = request.json()
37 html = HTML(
38 html=json_response["items_html"], url="bunk", default_encoding="utf-8"
~/opt/miniconda3/lib/python3.8/site-packages/requests/models.py in json(self, **kwargs)
896 # used.
897 pass
--> 898 return complexjson.loads(self.text, **kwargs)
899
900 @property
~/opt/miniconda3/lib/python3.8/json/__init__.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
355 parse_int is None and parse_float is None and
356 parse_constant is None and object_pairs_hook is None and not kw):
--> 357 return _default_decoder.decode(s)
358 if cls is None:
359 cls = JSONDecoder
~/opt/miniconda3/lib/python3.8/json/decoder.py in decode(self, s, _w)
335
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
339 if end != len(s):
~/opt/miniconda3/lib/python3.8/json/decoder.py in raw_decode(self, s, idx)
353 obj, end = self.scan_once(s, idx)
354 except StopIteration as err:
--> 355 raise JSONDecodeError("Expecting value", s, err.value) from None
356 return obj, end
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 26
- Comments: 21 (1 by maintainers)
I have the same issue on Python 3.6, Google colab.
I have the same issue on Python 3.8.2. Everyone know other library for scrapper or know If the new API twitter version allows to get tweets from more than seven days ago? Thanks a lot.
Everything is very simple - we take JSON from the link, take from it the “body” field with the timeline and parse it as usual. JSON also has “minPosition” and “maxPosition” headers for pagination. Implemented in my Go project https://github.com/n0madic/twitter-scraper.
Found another source of tweets - widgets at https://publish.twitter.com/ Get JSON from
https://syndication.twitter.com/timeline/profile?screen_name=<NAME>
The information there is incomplete, for example, there are no likes, retweets, etc. But there is no rate limit.Here is my attempt at a solution based on @Foo-Manroot instructions. https://gist.github.com/dgnsrekt/8cdb0c10e4a8fe6dcef65ee3a66a4e33 I’m having an issue with getting rate limited. Once, I can get past that issue I can make it nice and pretty and submit a pull request.
If anyone interested in a workaround, please check this out @yimingStar @bisguzar
https://github.com/Foo-Manroot/tweet-feed/issues/1#issuecomment-686635709
Dont know if this method is good enough.
It is because the url request for the user profile is somehow unavailable. See issue #166, the user-agent that this module is using cannot get the proper response now. Maybe Twitter has banned the user-agent, or maybe Twitter has changed all their website into Client Side Rendering causing request-html failed to get the elements that it usually could.