yt-dlp: [Twitter] Account is now required to download (Failed to parse JSON)

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

I’m reporting that yt-dlp is broken on a supported site
I’ve verified that I’m running yt-dlp version 2023.06.22 (update instructions) or later (specify commit)
I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
I’ve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
I’ve read the guidelines for opening an issue
I’ve read about sharing account credentials and I’m willing to share it if required

Region

United States

Provide a description that is worded well enough to be understood

As of today or yesterday, Twitter now requires an account to view tweets. As such, yt-dlp now breaks if username+password or cookies are not specified. I’m afraid there won’t be any fix to this, and we’ll have to make it error out when credentials are not specified.

Provide verbose output that clearly demonstrates the problem

Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
If using API, add 'verbose': True to YoutubeDL params instead
Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-vU', 'https://twitter.com/Twitter/status/1263145271946551300']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2023.06.30.182724 [af1fd12f6] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19044-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg 2021-08-25-git-e41bd075dd-full_build-www.gyan.dev (setts), ffprobe 2021-08-25-git-e41bd075dd-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1854 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Available version: nightly@2023.06.30.182724, Current version: nightly@2023.06.30.182724
Current Build Hash: d11828df364bc4c5053fe96b816ebf614a81cbcf1491e12544d4853a2ed355dc
yt-dlp is up to date (nightly@2023.06.30.182724)
[twitter] Extracting URL: https://twitter.com/Twitter/status/1263145271946551300
[twitter] 1263145271946551300: Downloading guest token
[twitter] 1263145271946551300: Downloading GraphQL JSON
ERROR: [twitter] 1263145271946551300: 1263145271946551300: Failed to parse JSON (caused by JSONDecodeError("Expecting value in '': line 1 column 1 (char 0)")); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 708, in extract
  File "yt_dlp\extractor\twitter.py", line 1148, in _real_extract
  File "yt_dlp\extractor\twitter.py", line 326, in _call_graphql_api
  File "yt_dlp\extractor\twitter.py", line 303, in _call_api
  File "yt_dlp\extractor\common.py", line 1062, in download_content
  File "yt_dlp\extractor\common.py", line 1032, in download_handle
  File "yt_dlp\extractor\common.py", line 1022, in parse
  File "yt_dlp\extractor\common.py", line 1009, in _parse_json
  File "yt_dlp\extractor\common.py", line 992, in __print_error

  File "yt_dlp\utils\_utils.py", line 624, in decode
  File "json\decoder.py", line 337, in decode
  File "json\decoder.py", line 355, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "yt_dlp\extractor\common.py", line 1006, in _parse_json
  File "json\__init__.py", line 370, in loads
  File "yt_dlp\utils\_utils.py", line 632, in decode
json.decoder.JSONDecodeError: Expecting value in '': line 1 column 1 (char 0)

About this issue

Original URL
State: closed
Created a year ago
Reactions: 13
Comments: 17 (5 by maintainers)

Commits related to this issue

[extractor/twitter] Fix unauthenticated extraction (#7476) Closes #7473 Authored by: bashonly — committed to yt-dlp/yt-dlp by bashonly a year ago

Most upvoted comments

Here’s some insight to the (reported) cause for this change:

Off topic

Twitter failed to pay it’s $70 million in outstanding payments to AWS
The AWS contract expired on June 30th, and Twitter has lost a large amount of its capacity and must rely on Google Cloud for the vast majority of it’s hosting

They attempted to mitigate this yesterday by disabling access to anyone not signed in
This failed as embedded tweets and 3rd party services just kept trying again and again to load tweets
Today a rate limit was implemented for the ability to read tweets (300 new non-verified, 600 non-verified, 6000 verified)

It’s not clear whether this will be a permanent change, but there is a chance that it will be reversed if they can fix their infrastructure problems.

+23

ayesc9000 on Jul 3, 2023

So it is at least possible to download with yt-dlp with an account at all? Thanks for the clarification.

You can download with or without an account.

To download without an account, you’ll need to be using the nightly build (run yt-dlp --update-to nightly) or a master branch installation.

To download with an account, you can pass cookies as the comment above explains

bashonly on Sep 16, 2023

Is there a way to download that version if I installed yt-dlp on windows through pip?

pip install --force-reinstall https://github.com/bashonly/yt-dlp/archive/fix/twitter-noauth.tar.gz

pukkandan on Jul 4, 2023

This is place to discuss the the issue in relation to yt-dlp. Take tangential discussions elsewhere.

pukkandan on Jul 3, 2023

Log when credentials are specified:

[debug] Command-line config: ['-vU', '-u', 'PRIVATE', '-p', 'PRIVATE', 'https://twitter.com/Twitter/status/1263145271946551300']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2023.06.30.182724 [af1fd12f6] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19044-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg 2021-08-25-git-e41bd075dd-full_build-www.gyan.dev (setts), ffprobe 2021-08-25-git-e41bd075dd-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1854 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Available version: nightly@2023.06.30.182724, Current version: nightly@2023.06.30.182724
Current Build Hash: d11828df364bc4c5053fe96b816ebf614a81cbcf1491e12544d4853a2ed355dc
yt-dlp is up to date (nightly@2023.06.30.182724)
[twitter] Requesting cookies
[twitter] Downloading guest token
[twitter] Downloading flow token
[twitter] Submitting JS instrumentation response
[twitter] Submitting username
[twitter] Submitting password
[twitter] Submitting account duplication check
[twitter] Logging in
[twitter] Extracting URL: https://twitter.com/Twitter/status/1263145271946551300
[twitter] 1263145271946551300: Downloading GraphQL JSON
[debug] [twitter] Extracting from video info: 1263145212760805376
[twitter] 1263145271946551300: Downloading m3u8 information
[debug] Sort order given by extractor: res, br, size, proto
[debug] Formats sorted by: hasvid, ie_pref, res, br, size, proto, lang, quality, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, asr, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 1263145212760805376: Downloading 1 format(s): http-2176
[debug] Invoking http downloader on "https://video.twimg.com/amplify_video/1263145212760805376/vid/1280x720/9jous8HM0_duxL0w.mp4?tag=13"
[debug] File locking is not supported. Proceeding without locking
[download] Destination: Twitter - Testing, testing...  A new way to have a convo with exactly who you want. We’re starting with a small % globally, so keep your 👀 out to see it in action. [1263145212760805376].mp4
[download] 100% of    3.35MiB in 00:00:01 at 2.91MiB/s

ztgasdf on Jun 30, 2023

@dmovztoq you can do it easily by logging into your account in firefox (then you have to close firefox once), and then downloading the video as such (assuming you are on ubuntu and have firefox pre-installed with snap):

yt-dlp --cookies-from-browser firefox:~/snap/firefox/common/.mozilla/firefox/ <tweet link>

dimisjim on Sep 16, 2023

On twitter someone speculated it was that they failed to pay for Google services, not aws. In any case this just goes to show Elon is even dumber than even his worst critics suggested he is.

The new Twitter CEO fixed up their relationship with Google near the end of July, and paid off their debts to them. (According to Bloomberg)

Regardless, you’re totally correct that this is incredibly stupid. It’s a shame that Elon fired the people responsible to make sure he doesn’t make stupid decisions just because he didn’t like that they told him his ideas were bad. I’ve never seen a CEO think that driving their users away from their platform is a good idea.

ayesc9000 on Jul 3, 2023