yt-dlp: [twitter] Unable to download JSON metadata (only age-restricted links)

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

  • I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

  • I’m reporting a bug unrelated to a specific site
  • I’ve verified that I’m running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
  • I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
  • I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
  • I’ve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
  • I’ve read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

came across this link here: https://twitter.com/_Saraurora_/status/1614746772974501896

Seems to be “age restricted”, cause if not logged in it shows this message on the twitter page: Age-restricted adult content. This content might not be appropriate for people under 18 years old. To view this media, you’ll need to log in to Twitter.

Could not test this with --cookies though, due to limited internet access at the moment.

all other public links download just fine.

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • If using API, add 'verbose': True to YoutubeDL params instead
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

./yt-dlp -Uv -F "https://twitter.com/_Saraurora_/status/1614746772974501896"
[debug] Command-line config: ['-Uv', '-F', 'https://twitter.com/_Saraurora_/status/1614746772974501896']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.03 [934496428] (zip)
[debug] Python 3.11.3 (CPython arm64 64bit) - macOS-13.3-arm64-arm-64bit (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: phantomjs 2.1.1
[debug] Optional libraries: sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Extractor Plugins: SamplePluginIE
[debug] Post-Processor Plugins: SamplePluginPP
[debug] Loaded 1852 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.03
Current Build Hash: cfa97267ed664425bb74e861d3da044182bf65a8704998146da441896c922562
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp/releases/latest/download/_update_spec
Updating to stable@2023.03.04 ...
[debug] Downloading yt-dlp from https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp
[debug] Downloading SHA2-256SUMS from https://github.com/yt-dlp/yt-dlp/releases/latest/download/SHA2-256SUMS
Updated yt-dlp to stable@2023.03.04
[debug] Restarting: /opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python ./yt-dlp -Uv -F https://twitter.com/_Saraurora_/status/1614746772974501896
[debug] Command-line config: ['-Uv', '-F', 'https://twitter.com/_Saraurora_/status/1614746772974501896']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (zip)
[debug] Python 3.11.3 (CPython arm64 64bit) - macOS-13.3-arm64-arm-64bit (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: phantomjs 2.1.1
[debug] Optional libraries: no_Cryptodome-None, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Extractor Plugins: SamplePluginIE
[debug] Post-Processor Plugins: SamplePluginPP
[debug] Loaded 1787 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
Current Build Hash: 91cad9f121c1f6f0a81b747415c46ecba0ff331ed38cc6433040b4ac7b6e15ca
yt-dlp is up to date (stable@2023.03.04)
[twitter] Extracting URL: https://twitter.com/_Saraurora_/status/1614746772974501896
[twitter] 1614746772974501896: Downloading guest token
[twitter] 1614746772974501896: Downloading JSON metadata
WARNING: [twitter] Twitter API gave 404 response, retrying with deprecated auth token. Only one media item can be extracted
[twitter] 1614746772974501896: Downloading guest token
[twitter] 1614746772974501896: Downloading JSON metadata
ERROR: [twitter] 1614746772974501896: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "./yt-dlp/yt_dlp/extractor/common.py", line 694, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "./yt-dlp/yt_dlp/extractor/twitter.py", line 974, in _real_extract
    status = traverse_obj(self._call_api(f'statuses/show/{twid}.json', twid, {
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "./yt-dlp/yt_dlp/extractor/twitter.py", line 136, in _call_api
    raise last_error
  File "./yt-dlp/yt_dlp/extractor/twitter.py", line 130, in _call_api
    result = self._download_json(
             ^^^^^^^^^^^^^^^^^^^^
  File "./yt-dlp/yt_dlp/extractor/common.py", line 1048, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "./yt-dlp/yt_dlp/extractor/common.py", line 1012, in download_handle
    res = self._download_webpage_handle(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "./yt-dlp/yt_dlp/extractor/common.py", line 882, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "./yt-dlp/yt_dlp/extractor/common.py", line 839, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "./yt-dlp/yt_dlp/extractor/common.py", line 821, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "./yt-dlp/yt_dlp/YoutubeDL.py", line 3742, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 31 (14 by maintainers)

Commits related to this issue

Most upvoted comments

Twitter broke our bypass for extracting age-restricted tweets without auth.

You’ll need to pass login cookies:

$ yt-dlp -F "https://twitter.com/_Saraurora_/status/1614746772974501896" --cookies-from-browser "firefox:xu2iz3kn.default-release::twitter"
[Cookies] Extracting cookies from firefox
[Cookies] Extracted 61 cookies from firefox
[twitter] Extracting URL: https://twitter.com/_Saraurora_/status/1614746772974501896
[twitter] 1614746772974501896: Downloading JSON metadata
[twitter] 1614746772974501896: Downloading m3u8 information
[info] Available formats for 1614746727411499011:
ID        EXT RESOLUTION │   FILESIZE   TBR PROTO │ VCODEC        VBR ACODEC    ABR
───────────────────────────────────────────────────────────────────────────────────
hls-256   mp4 600x270    │ ~316.51KiB  256k m3u8  │ avc1.4d001e  256k mp4a.40.2  0k
http-256  mp4 600x270    │ ~316.51KiB  256k https │ unknown      256k unknown    0k
hls-832   mp4 800x360    │ ~  1.00MiB  832k m3u8  │ avc1.4d001f  832k mp4a.40.2  0k
http-832  mp4 800x360    │ ~  1.00MiB  832k https │ unknown      832k unknown    0k
hls-2176  mp4 1280x576   │ ~  2.63MiB 2176k m3u8  │ avc1.640020 2176k mp4a.40.2  0k
http-2176 mp4 1280x576   │ ~  2.63MiB 2176k https │ unknown     2176k unknown    0k

The extractor should be updated to catch this error and raise_login_required. And I think the deprecated token code can be removed now since it no longer serves a purpose

yt-dlp -F "https://twitter.com/someagerestrictedlink" --cookies-from-browser firefox

Works for me

I don’t know how to use --cookies and chat GPT doesn’t know either.

If the only issue is that you can’t write a file, you could just pass all the cookies in the Cookie header formatted correctly. If you are cherry picking specific ones, it’s on you to figure out which ones are relevant.

@flashthepublic Because of the code below, the extractor will never treat you as logged in unless you pass --cookies or --cookies-from-browser: https://github.com/yt-dlp/yt-dlp/blob/c3f624ef0a5d7a6ae1c5ffeb243087e9fc7d79dc/yt_dlp/extractor/twitter.py#L91-L93 And if not logged in, the extractor fetches a guest token and passes it as a header with API requests, which will most likely interfere with authentication.

EDIT: see pukkandan’s comment below

From looking at the code the only cookie used is ct0

There are other cookies that are required (e.g. auth_token), but the extractor looks in the cookiejar for them, not the added headers. Don’t manually add cookie headers, just pass your auth cookies with --cookies or --cookies-from-browser