youtube-dl: Whether https proxy `--proxy=https://user:pass@a.b.com` is not supported? [it isn't]

Checklist

  • I’m asking a question
  • I’ve looked through the README and FAQ for similar questions
  • I’ve searched the bugtracker for similar questions including closed ones

Question

I’ve checked that https proxy with username authenticated can not work well, for example https://user:pass@a.b.com will output ‘Connection reset by peer’.

image

for security, proxy address can not be provided here.

I’m using macOS to run this program.

Do I wrongly setup somewhere?

About this issue

  • Original URL
  • State: open
  • Created 10 months ago
  • Comments: 20 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Eg:

--- old/youtube-dl/youtube_dl/utils.py
+++ new/youtube-dl/youtube_dl/utils.py
@@ -5833,10 +5833,13 @@
 
         if proxy == '__noproxy__':
             return None  # No Proxy
-        if compat_urllib_parse.urlparse(proxy).scheme.lower() in ('socks', 'socks4', 'socks4a', 'socks5'):
+        scheme = compat_urllib_parse.urlparse(proxy).scheme.lower()
+        if scheme in ('socks', 'socks4', 'socks4a', 'socks5'):
             req.add_header('Ytdl-socks-proxy', proxy)
             # youtube-dl's http/https handlers do wrapping the socket with socks
             return None
+        elif scheme == 'https':
+        	raise NotImplementedError('''urllib2 doesn't support https proxies''')
         return compat_urllib_request.ProxyHandler.proxy_open(
             self, req, proxy, type)
+2
dirkf on Sep 20, 2023

Maybe:

-        	raise NotImplementedError('''urllib2 doesn't support https proxies''')
+        	err = NotImplementedError('''urllib2 doesn't support https proxies''')
+        	raise ExtractorError('Proxy error', cause=err, expected=True)

I suppose the message also appears for PROXY environment vars?

+1
dirkf on Sep 21, 2023

Thanks, it works as expected when latest nightly build has been patched accordingly:

youtube-dl --proxy "https://ProxyHostName:443" -vF "Fh7UgrHMjbQ" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--proxy', 'https://ProxyHostName:443', '-vF', 'Fh7UgrHMjbQ']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.07 [66ab0814c] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {'https': 'https://ProxyHostName:443', 'http': 'https://ProxyHostName:443'}
[youtube] Fh7UgrHMjbQ: Downloading webpage
Traceback (most recent call last):
  File "__main__.py", line 19, in <module>
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 473, in main
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 463, in _real_main
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 2223, in download
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 856, in extract_info
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 863, in wrapper
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 959, in __extract_info
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 565, in extract
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 1850, in _real_extract
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 832, in _download_webpage
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 698, in _download_webpage_handle
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 665, in _request_webpage
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 2465, in urlopen
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 464, in open
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 482, in _open
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 442, in _call_chain
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 735, in <lambda>
  File "utils.py", line 5843, in proxy_open
NotImplementedError: urllib2 doesn't support https proxies
+1
Vangelis66 on Sep 21, 2023

It’s a pity, though, we haven’t heard anything additional from OP 😉 …

… At long last, yesterday I got my hands on a US HTTPS proxy server with authentication, so I can now conduct some tests myself on the yt-dl issue at hand…

Youtube title/id 7LBYrDx_784 is BLOCKED in my location (purportedly due to “rights” reasons), but is available in the US…

Trying a DIRECT fetch command yields the uninformative message This video is not available:

No Proxy 
yt-dl -vF "7LBYrDx_784" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '7LBYrDx_784']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: ffmpeg n6.1-dev-1945-N-111829-g3c9dc0, ffprobe n6.1-dev-1945-N-111829-g3c9dc0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] 7LBYrDx_784: Downloading webpage
ERROR: Video unavailable
This video is not available
Traceback (most recent call last):
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 859, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 955, in __extract_info
    ie_result = ie.extract(url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 565, in extract
    ie_result = self._real_extract(url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 2120, in _real_extract
    raise ExtractorError(reason, expected=True)
youtube_dl.utils.ExtractorError: Video unavailable
This video is not available

Trying the HTTPS proxy via the --proxy cmdline option (use case reported by OP):

--proxy cmdline option 

yt-dl --proxy "https://ProxyUserName:ProxyPassword@77.*.*.*:443" -vF "7LBYrDx_784" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '--proxy', 'https://ProxyUserName:ProxyPassword@77.*.*.*:443', '-vF', '7LBYrDx_784']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: ffmpeg n6.1-dev-1945-N-111829-g3c9dc0, ffprobe n6.1-dev-1945-N-111829-g3c9dc0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {'https': 'https://ProxyUserName:ProxyPassword@77.*.*.*:443', 'http': 'https://ProxyUserName:ProxyPassword@77.*.*.*:443'}
[youtube] 7LBYrDx_784: Downloading webpage
WARNING: Unable to download webpage: ''
[youtube] 7LBYrDx_784: Downloading API JSON
ERROR: Unable to download API page: '' (caused by BadStatusLine("''",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 665, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 2461, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Python3410-32\lib\urllib\request.py", line 464, in open
    response = self._open(req, data)
  File "C:\Python3410-32\lib\urllib\request.py", line 482, in _open
    '_open', req)
  File "C:\Python3410-32\lib\urllib\request.py", line 442, in _call_chain
    result = func(*args)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\utils.py", line 2889, in https_open
    req, **kwargs)
  File "C:\Python3410-32\lib\urllib\request.py", line 1183, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "C:\Python3410-32\lib\http\client.py", line 1137, in request
    self._send_request(method, url, body, headers)
  File "C:\Python3410-32\lib\http\client.py", line 1182, in _send_request
    self.endheaders(body)
  File "C:\Python3410-32\lib\http\client.py", line 1133, in endheaders
    self._send_output(message_body)
  File "C:\Python3410-32\lib\http\client.py", line 963, in _send_output
    self.send(msg)
  File "C:\Python3410-32\lib\http\client.py", line 898, in send
    self.connect()
  File "C:\Python3410-32\lib\http\client.py", line 1279, in connect
    super().connect()
  File "C:\Python3410-32\lib\http\client.py", line 874, in connect
    self._tunnel()
  File "C:\Python3410-32\lib\http\client.py", line 852, in _tunnel
    (version, code, message) = response._read_status()
  File "C:\Python3410-32\lib\http\client.py", line 356, in _read_status
    raise BadStatusLine(line)

the ERROR produced is:

ERROR: Unable to download API page: '' (caused by BadStatusLine("''",))

IOW, the connection to the HTTPS proxy cannot be established 😭 …

Sadly, I can also confirm that specifying the HTTPS proxy via the HTTPS_PROXY env var doesn’t work 😞 either (as reported in the “downstream” issue):

HTTPS_PROXY env var 
set HTTPS_PROXY=https://ProxyUserName:ProxyPassword@77.*.*.*:443 && yt-dl -vF "7LBYrDx_784" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '7LBYrDx_784']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: ffmpeg n6.1-dev-1945-N-111829-g3c9dc0, ffprobe n6.1-dev-1945-N-111829-g3c9dc0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {'https': 'https://ProxyUserName:ProxyPassword@77.*.*.*:443'}
[youtube] 7LBYrDx_784: Downloading webpage
WARNING: Unable to download webpage: ''
[youtube] 7LBYrDx_784: Downloading API JSON
ERROR: Unable to download API page: '' (caused by BadStatusLine("''",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 665, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 2461, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Python3410-32\lib\urllib\request.py", line 464, in open
    response = self._open(req, data)
  File "C:\Python3410-32\lib\urllib\request.py", line 482, in _open
    '_open', req)
  File "C:\Python3410-32\lib\urllib\request.py", line 442, in _call_chain
    result = func(*args)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\utils.py", line 2889, in https_open
    req, **kwargs)
  File "C:\Python3410-32\lib\urllib\request.py", line 1183, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "C:\Python3410-32\lib\http\client.py", line 1137, in request
    self._send_request(method, url, body, headers)
  File "C:\Python3410-32\lib\http\client.py", line 1182, in _send_request
    self.endheaders(body)
  File "C:\Python3410-32\lib\http\client.py", line 1133, in endheaders
    self._send_output(message_body)
  File "C:\Python3410-32\lib\http\client.py", line 963, in _send_output
    self.send(msg)
  File "C:\Python3410-32\lib\http\client.py", line 898, in send
    self.connect()
  File "C:\Python3410-32\lib\http\client.py", line 1279, in connect
    super().connect()
  File "C:\Python3410-32\lib\http\client.py", line 874, in connect
    self._tunnel()
  File "C:\Python3410-32\lib\http\client.py", line 852, in _tunnel
    (version, code, message) = response._read_status()
  File "C:\Python3410-32\lib\http\client.py", line 356, in _read_status
    raise BadStatusLine(line)

Thus, I’ve hidden my previous suggestion above …

One quick workaround would be to use a network repeater

One such application I’m familiar with from past usage is GOST (GoSecureTunnel):

https://github.com/ginuerzh/gost

The app (written in golang) has seen more recent development compared to stone referenced in the “downstream” issue tracker 😉 ; gost can be configured to launch a local HTTP proxy (that yt-dl has support for and can be pointed to) and, at the same time, forward the requests made through that proxy to a remote HTTPS proxy; gost has no problem authenticating to that remote HTTPS proxy, if basic authentication has been enforced on the proxy; in a nutshell, the command to use would look like below:

gost -L="http://localhost:8080" -F="https://ProxyUserName:ProxyPassword@ProxyHostname(|ProxyIP):ProxyPort"

Indeed, in my testing I proved it works as advertised 👍 :

Using gost as MITM 
(first cmdline window, shall remain open for the duration)
gost -L="http://localhost:8080" -F="https://ProxyUserName:ProxyPassword@77.*.*.*:443"

(second cmdline window)
yt-dl --proxy "http://localhost:8080" -vF "7LBYrDx_784" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '--proxy', 'http://localhost:8080', '-vF', '7LBYrDx_784']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: ffmpeg n6.1-dev-1945-N-111829-g3c9dc0, ffprobe n6.1-dev-1945-N-111829-g3c9dc0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {'https': 'http://localhost:8080', 'http': 'http://localhost:8080'}
[youtube] 7LBYrDx_784: Downloading webpage
[debug] [youtube] Decrypted nsig -Sq7NdL_EDAFmZ_X => vQkYN_oLLglXcA
[debug] [youtube] Decrypted nsig pTv43VKeQKiF6Nma => uz32DjmqPXMKhA
[info] Available formats for 7LBYrDx_784:
format code  extension  resolution note
249          webm       audio only tiny   57k , webm_dash container, opus  (48000Hz), 897.81KiB
250          webm       audio only tiny   76k , webm_dash container, opus  (48000Hz), 1.17MiB
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 1.98MiB
251          webm       audio only tiny  151k , webm_dash container, opus  (48000Hz), 2.31MiB
160          mp4        146x144    144p    9k , mp4_dash container, avc1.4d400b, 25fps, video only, 151.79KiB
278          webm       146x144    144p   21k , webm_dash container, vp9, 25fps, video only, 330.81KiB
394          mp4        146x144    144p   21k , mp4_dash container, av01.0.00M.08, 25fps, video only, 332.29KiB
133          mp4        244x240    240p   13k , mp4_dash container, avc1.4d400c, 25fps, video only, 209.02KiB
395          mp4        244x240    240p   29k , mp4_dash container, av01.0.00M.08, 25fps, video only, 468.01KiB
242          webm       244x240    240p   38k , webm_dash container, vp9, 25fps, video only, 606.50KiB
134          mp4        366x360    360p   23k , mp4_dash container, avc1.4d4015, 25fps, video only, 364.02KiB
396          mp4        366x360    360p   44k , mp4_dash container, av01.0.00M.08, 25fps, video only, 701.91KiB
243          webm       366x360    360p   62k , webm_dash container, vp9, 25fps, video only, 979.65KiB
135          mp4        488x480    480p   38k , mp4_dash container, avc1.4d401e, 25fps, video only, 608.24KiB
397          mp4        488x480    480p   65k , mp4_dash container, av01.0.01M.08, 25fps, video only, 1.00MiB
244          webm       488x480    480p   99k , webm_dash container, vp9, 25fps, video only, 1.52MiB
136          mp4        732x720    720p   70k , mp4_dash container, avc1.4d401f, 25fps, video only, 1.08MiB
398          mp4        732x720    720p  124k , mp4_dash container, av01.0.04M.08, 25fps, video only, 1.90MiB
247          webm       732x720    720p  200k , webm_dash container, vp9, 25fps, video only, 3.07MiB
18           mp4        366x360    360p   71k , avc1.42001E, 25fps, mp4a.40.2 (22050Hz) (best)

… and this is what the gost (first) window prints:

2023/09/17 06:47:06 route.go:695: http://localhost:8080 on 127.0.0.1:8080
2023/09/17 06:48:46 http.go:161: [http] 127.0.0.1:13561 -> http://localhost:8080 -> www.youtube.com:443
2023/09/17 06:48:46 http.go:256: [route] 127.0.0.1:13561 -> http://localhost:8080 -> 1@https://77.*.*.*:443 -> www.youtube.com:443
2023/09/17 06:48:47 http.go:311: [http] 127.0.0.1:13561 <-> www.youtube.com:443
2023/09/17 06:48:48 http.go:313: [http] 127.0.0.1:13561 >-< www.youtube.com:443

Since OP is on MacOS, gost does come in precompiled binaries for that platform:

https://github.com/ginuerzh/gost/releases/download/v2.11.5/gost-darwin-amd64-2.11.5.gz
https://github.com/ginuerzh/gost/releases/download/v2.11.5/gost-darwin-arm64-2.11.5.gz

But there’s not likely to be a back-port of that because of dependencies

… Is this going to be a won't/can't fix then? Could a “requests” yt-dl branch be created and maintained for those able to run yt-dl on py3.7+ ???

+1
Vangelis66 on Sep 18, 2023

https://github.com/yt-dlp/yt-dlp/issues/1890 is very useful, these especially.

From https://github.com/yt-dlp/yt-dlp/issues/1890#issuecomment-986245020:

Can confirm, HTTP proxies work but HTTPS ones are broken, whether via --proxy or environment variable.

From https://github.com/yt-dlp/yt-dlp/issues/1890#issuecomment-986977600:

One quick workaround would be to use a network repeater like stone (https://www.gcd.org/sengoku/stone/) that is capable of translation such as:

[python]<--(plain-text proxy)-->[repeater]<--(TLS proxy)-->[secure web (HTTPS) proxy server]

Run stone as

$ stone uk2344.nordvpn.com:89/ssl 8080

and then invoke yt-dlp in another terminal as

$ yt-dlp --proxy "http://xxxxxxxxx%40xxxxxxx.com:xxxxxxxxxxxxxxxx@localhost:8080" "https://www.youtube.com/watch?v=Fh7UgrHMjbQ" [confirmed working with sufficiently recent OpenSSL].

According to https://github.com/yt-dlp/yt-dlp/issues/1890#issuecomment-1162533509, https://github.com/yt-dlp/yt-dlp/pull/3668 adds support in yt-dlp.

But there’s not likely to be a back-port of that because of dependencies that would mean you could just run yt-dlp anyway.

+1
dirkf on Sep 12, 2023

https proxies are not supported by either youtube-dl or yt-dlp (yet). Due to a bug in Python, they are treated as http in youtube-dl.

related: https://github.com/yt-dlp/yt-dlp/issues/1890 https://github.com/ytdl-org/youtube-dl/issues/26709

+1
coletdjnz on Sep 11, 2023

“yt-dl” is just an abbreviation for “youtube-dl”.

+1
dirkf on Sep 10, 2023

@Vangelis66 I’m calling youtube-dl not yt-dl. Do I use wrong command? I will try to install yt-dl to test. thank you.

+1
wisetc on Sep 10, 2023

@wisetc : Can you not set your HTTPS proxy (with basic authentication) as an env var in your terminal?

export HTTP_PROXY="http://ProxyUserName:ProxyPassword@ProxyHostname:ProxyPort" 

export HTTPS_PROXY="https://ProxyUserName:ProxyPassword@ProxyHostname:ProxyPort"

Then, yt-dl in that same terminal instance should use, by default, the proxied connection 😉 …

Later edit: Tested by myself (on Windows 😜 ) as NOT WORKING 😠 😢 ; read below …

+1
Vangelis66 on Sep 18, 2023
Read more comments on GitHub
← youtube-dl: What does "Unable to extract uploader id" mean, and how do I update YouTube-dl?
youtube-dl: Why cant i download? →