yt-dlp: [tik-tok] ERROR: [generic] Unable to download webpage: The read operation timed out
Checklist
- I’m reporting a bug unrelated to a specific site
- I’ve verified that I’m running yt-dlp version 2021.12.27. (update instructions)
- I’ve checked that all provided URLs are alive and playable in a browser
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
- I’ve read the guidelines for opening an issue
Description
Cant download tik-tok video: WARNING: [generic] Could not send HEAD request ERROR: [generic] Unable to download webpage: The read operation timed out (caused by timeout(‘The read operation timed out’,)). ytdl-patched version 2022.01.19.114514 doesn’t work also.
Verbose log
yt-dlp -Uv -o ./abc.mp4 https://vm.tiktok.com/ZSe4FqkKd
[debug] Command-line config: ['-Uv', '-o', './abc.mp4', 'https://vm.tiktok.com/ZSe4FqkKd']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, err UTF-8, pref UTF-8
[debug] yt-dlp version 2021.12.27 [6223f67a8] (zip)
[debug] Python version 3.6.9 (CPython 64bit) - Linux-4.15.0-156-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.8, ffprobe 3.4.8
[debug] Optional libraries: Crypto, secretstorage, sqlite
[debug] Proxy map: {}
Latest version: 2021.12.27, Current version: 2021.12.27
yt-dlp is up to date (2021.12.27)
[debug] [generic] Extracting URL: https://vm.tiktok.com/ZSe4FqkKd
[generic] ZSe4FqkKd: Requesting header
WARNING: [generic] Could not send HEAD request to https://vm.tiktok.com/ZSe4FqkKd: The read operation timed out
[generic] ZSe4FqkKd: Downloading webpage
ERROR: [generic] Unable to download webpage: The read operation timed out (caused by timeout('The read operation timed out',)); please report this issue on https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; type yt-dlp -U to update. Be sure to call yt-dlp with the --verbose flag and include its complete output. (caused by timeout('The read operation timed out',)); please report this issue on https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; type yt-dlp -U to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 717, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3458, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.6/urllib/request.py", line 564, in error
result = self._call_chain(*args)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 756, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.6/urllib/request.py", line 564, in error
result = self._call_chain(*args)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 756, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/bin/yt-dlp/yt_dlp/utils.py", line 1394, in https_open
req, **kwargs)
File "/usr/lib/python3.6/urllib/request.py", line 1328, in do_open
r = h.getresponse()
File "/usr/lib/python3.6/http/client.py", line 1377, in getresponse
response.begin()
File "/usr/lib/python3.6/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.6/http/client.py", line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.6/ssl.py", line 874, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.6/ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 9
- Comments: 24 (7 by maintainers)
Seems to be a regression, I cannot reproduce on release (2021.12.27) Using
--user-agent facebookexternalhit/1.1works as a workaroundFor anyone having issues spoofing the user-agent in embedded yt-dlp, setting the standard header before requesting the urls works for me:
@pukkandan I seem to be still getting the exact same error after force reinstalling from master branch. Any ideas? I’m assuming that commit is supposed to fix it
Oke. The issue seems to be that yt_dlp uses HTTP 1.1, which TikTok rejects for certain headers. (e.g. the default one which simulates the Chrome header). Using HTTP 2 instead fixes the issue (when using cURL). See following script:
Well it is one way, but when handling multiple requests at once using multiple
YoutubeDLinstances in an multithreaded embedded environment makes the global setting not the best way to go forward.When using
YoutubeDL.extract_infoone can provide anextra_infodictionary which is used to handle the download request. One of the parameters that can be provided with that ishttp_headers. I use it explicitly to provide aRefererheader to the video download. Sadly this additional dict is only important when it comes to downloading the actual video file, not for getting the redirections chain.Something to initialize the
YoutubeDLclass that would allow one to configure the headers used further on would be nice as globalstd_headersisn’t always possible to use.I think it may have something to do with HTTP/2. I tried
requests(allow_redirects=False) andurllib.request.urlopenon the HTML page, but they both return a 200 status code, instead of a 301 redirect.I tried with the
httpxlibrary, and it properly returns a 301 status code, and from there I can then get the next location (the www.tiktok.com location) from the location header of the request. Or, I can just pass the vm.tiktok.com link tohttpx.get(url, follow_redirects=True)and get the www.tiktok.com link that way.@pukkandan Seems to be an issue with the redirect.
https://vm.tiktok.com/TTPdr42PwF/doesn’t work, buthttps://www.tiktok.com/@jangyhayden/video/7055075515409059119works fine.It also seems that
https://vm.tiktok.com/TTPdr42PwF/redirects tohttps://m.tiktok.com/v/7055075515409059119.html, which yt_dlp doesn’t work on either.My bad. Was testing the patched one. Here’s new log;