yt-dlp: [twitter] Unable to download JSON metadata (only age-restricted links)
DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
- I understand that I will be blocked if I intentionally remove or skip any mandatory* field
Checklist
- I’m reporting a bug unrelated to a specific site
- I’ve verified that I’m running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
- I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
- I’ve read the guidelines for opening an issue
Provide a description that is worded well enough to be understood
came across this link here: https://twitter.com/_Saraurora_/status/1614746772974501896
Seems to be “age restricted”, cause if not logged in it shows this message on the twitter page: Age-restricted adult content. This content might not be appropriate for people under 18 years old. To view this media, you’ll need to log in to Twitter.
Could not test this with --cookies though, due to limited internet access at the moment.
all other public links download just fine.
Provide verbose output that clearly demonstrates the problem
- Run your yt-dlp command with -vU flag added (
yt-dlp -vU <your command line>
) - If using API, add
'verbose': True
toYoutubeDL
params instead - Copy the WHOLE output (starting with
[debug] Command-line config
) and insert it below
Complete Verbose Output
./yt-dlp -Uv -F "https://twitter.com/_Saraurora_/status/1614746772974501896"
[debug] Command-line config: ['-Uv', '-F', 'https://twitter.com/_Saraurora_/status/1614746772974501896']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.03 [934496428] (zip)
[debug] Python 3.11.3 (CPython arm64 64bit) - macOS-13.3-arm64-arm-64bit (OpenSSL 1.1.1t 7 Feb 2023)
[debug] exe versions: phantomjs 2.1.1
[debug] Optional libraries: sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Extractor Plugins: SamplePluginIE
[debug] Post-Processor Plugins: SamplePluginPP
[debug] Loaded 1852 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.03
Current Build Hash: cfa97267ed664425bb74e861d3da044182bf65a8704998146da441896c922562
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp/releases/latest/download/_update_spec
Updating to stable@2023.03.04 ...
[debug] Downloading yt-dlp from https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp
[debug] Downloading SHA2-256SUMS from https://github.com/yt-dlp/yt-dlp/releases/latest/download/SHA2-256SUMS
Updated yt-dlp to stable@2023.03.04
[debug] Restarting: /opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python ./yt-dlp -Uv -F https://twitter.com/_Saraurora_/status/1614746772974501896
[debug] Command-line config: ['-Uv', '-F', 'https://twitter.com/_Saraurora_/status/1614746772974501896']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (zip)
[debug] Python 3.11.3 (CPython arm64 64bit) - macOS-13.3-arm64-arm-64bit (OpenSSL 1.1.1t 7 Feb 2023)
[debug] exe versions: phantomjs 2.1.1
[debug] Optional libraries: no_Cryptodome-None, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Extractor Plugins: SamplePluginIE
[debug] Post-Processor Plugins: SamplePluginPP
[debug] Loaded 1787 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
Current Build Hash: 91cad9f121c1f6f0a81b747415c46ecba0ff331ed38cc6433040b4ac7b6e15ca
yt-dlp is up to date (stable@2023.03.04)
[twitter] Extracting URL: https://twitter.com/_Saraurora_/status/1614746772974501896
[twitter] 1614746772974501896: Downloading guest token
[twitter] 1614746772974501896: Downloading JSON metadata
WARNING: [twitter] Twitter API gave 404 response, retrying with deprecated auth token. Only one media item can be extracted
[twitter] 1614746772974501896: Downloading guest token
[twitter] 1614746772974501896: Downloading JSON metadata
ERROR: [twitter] 1614746772974501896: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U
File "./yt-dlp/yt_dlp/extractor/common.py", line 694, in extract
ie_result = self._real_extract(url)
^^^^^^^^^^^^^^^^^^^^^^^
File "./yt-dlp/yt_dlp/extractor/twitter.py", line 974, in _real_extract
status = traverse_obj(self._call_api(f'statuses/show/{twid}.json', twid, {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./yt-dlp/yt_dlp/extractor/twitter.py", line 136, in _call_api
raise last_error
File "./yt-dlp/yt_dlp/extractor/twitter.py", line 130, in _call_api
result = self._download_json(
^^^^^^^^^^^^^^^^^^^^
File "./yt-dlp/yt_dlp/extractor/common.py", line 1048, in download_content
res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./yt-dlp/yt_dlp/extractor/common.py", line 1012, in download_handle
res = self._download_webpage_handle(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./yt-dlp/yt_dlp/extractor/common.py", line 882, in _download_webpage_handle
urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./yt-dlp/yt_dlp/extractor/common.py", line 839, in _request_webpage
raise ExtractorError(errmsg, cause=err)
File "./yt-dlp/yt_dlp/extractor/common.py", line 821, in _request_webpage
return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./yt-dlp/yt_dlp/YoutubeDL.py", line 3742, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 525, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 634, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 563, in error
return self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 643, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 31 (14 by maintainers)
Commits related to this issue
- [extractor/twitter] Default to GraphQL, handle auth errors (#6957) Closes #6763 Authored by: bashonly — committed to yt-dlp/yt-dlp by bashonly a year ago
Twitter broke our bypass for extracting age-restricted tweets without auth.
You’ll need to pass login cookies:
The extractor should be updated to catch this error and
raise_login_required
. And I think the deprecated token code can be removed now since it no longer serves a purposeWorks for me
I don’t know how to use --cookies and chat GPT doesn’t know either.
If the only issue is that you can’t write a file, you could just pass all the cookies in the
Cookie
header formatted correctly. If you are cherry picking specific ones, it’s on you to figure out which ones are relevant.@flashthepublic Because of the code below, the extractor will never treat you as logged in unless you pass
--cookies
or--cookies-from-browser
: https://github.com/yt-dlp/yt-dlp/blob/c3f624ef0a5d7a6ae1c5ffeb243087e9fc7d79dc/yt_dlp/extractor/twitter.py#L91-L93 And if not logged in, the extractor fetches a guest token and passes it as a header with API requests, which will most likely interfere with authentication.EDIT: see pukkandan’s comment below
There are other cookies that are required (e.g.
auth_token
), but the extractor looks in the cookiejar for them, not the added headers. Don’t manually add cookie headers, just pass your auth cookies with--cookies
or--cookies-from-browser