yt-dlp: [abciview] [zee5] ERROR: Data must be padded to 16 byte boundary in CBC mode
Checklist
- I’m reporting a bug unrelated to a specific site
- I’ve verified that I’m running yt-dlp version 2022.05.18 (update instructions) or later (specify commit)
- I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
- I’ve read the guidelines for opening an issue
Description
Download begins then stops with this error: “ERROR: Data must be padded to 16 byte boundary in CBC mode”.
Verbose log
[debug] Invoking hlsnative downloader on "https://iview-vod-hls.akamaized.net/prod/zw3/ZW3040A001S00MA1D1/20210908051741/hls/4500000/hdntl=exp=1653131813~acl=%2f*~data=hdntl~hmac=968b90b5a9ec2efc1e020f0d280af19668f4d9fff35fcde9c17d1236543497c6/stream.m3u8?aka_me_session_id=AAAAAAAAAAAlyohiAAAAAIV7zCwd0HGHqUdzBIZg0ja5vozQgVmfzpL8R6FzBHOiCKK1yd4SaW0I3Lpa2721IdAe29m1IaW5&aka_media_format_type=hls"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 270
[download] Destination: /mnt/AU/Temporary/NA/20210924-ZW3040A001S00-Series 1 Ep 1.mp4
[download] 0.4% of ~975.58MiB at 559.83KiB/s ETA 29:37[download] Got server HTTP error: Downloaded 3784784 bytes, expected 3788768 bytes. Retrying (attempt 1 of 10) ...
[download] 0.4% of ~975.58MiB at 373.69KiB/s ETA 44:22 (frag 1/270)ERROR: Data must be padded to 16 byte boundary in CBC mode
Traceback (most recent call last):
File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1395, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1479, in __extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1535, in process_ie_result
ie_result = self.process_video_result(ie_result, download=download)
File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2640, in process_video_result
self.process_info(new_info)
File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3130, in process_info
success, real_download = self.dl(temp_filename, info_dict)
File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2838, in dl
return fd.download(name, new_info, subtitle)
File "/usr/local/bin/yt-dlp/yt_dlp/downloader/common.py", line 445, in download
ret = self.real_download(filename, info_dict)
File "/usr/local/bin/yt-dlp/yt_dlp/downloader/hls.py", line 348, in real_download
return self.download_and_append_fragments(ctx, fragments, info_dict)
File "/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py", line 522, in download_and_append_fragments
decrypt_fragment(fragment, self._read_fragment(ctx)), fragment['frag_index'], ctx)
File "/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py", line 365, in decrypt_fragment
return unpad_pkcs7(aes_cbc_decrypt_bytes(frag_content, decrypt_info['KEY'], iv))
File "/usr/local/bin/yt-dlp/yt_dlp/aes.py", line 10, in aes_cbc_decrypt_bytes
return Cryptodome_AES.new(key, Cryptodome_AES.MODE_CBC, iv).decrypt(data)
File "/home/pi/.local/lib/python3.9/site-packages/Cryptodome/Cipher/_mode_cbc.py", line 246, in decrypt
raise ValueError("Data must be padded to %d byte boundary in CBC mode" % self.block_size)
ValueError: Data must be padded to 16 byte boundary in CBC mode
About this issue
- Original URL
- State: open
- Created 5 years ago
- Comments: 58 (13 by maintainers)
Commits related to this issue
- [downloader/hls] Do not resume download when stream uses AES-128 (#3810) — committed to ClosedPort22/yt-dlp by ClosedPort22 3 years ago
- [downloader/http] Detect content length mismatch when retrying (#3810) Detect and reject response bodies with an unexpected total length to prevent data corruption. Also fixed a regression where the... — committed to ClosedPort22/yt-dlp by ClosedPort22 3 years ago
- [fd/hls] Fix `BYTERANGE` logic (#11972) Closes #3578, Closes #3810, Closes #9400 Authored by: entourage8 — committed to saschazepter/yt-dlp by entourage8 2 years ago
- [fd/hls] Fix `BYTERANGE` logic (#11972) Closes #3578, Closes #3810, Closes #9400 Authored by: entourage8 — committed to subrat-lima/yt-dlp by entourage8 2 years ago
The author of #8081 identified this problem a while back:
(Note that using
--no-continueis not a valid solution currently, it only would be with that PR’s changes)IMO a better solution would be to make these changes:
--no-continueshould restart the download of the entire file, not just the fragment. It should also overwrite the temp file, not append to itHi all, just to advise I have spotted this issue on SBS, another Australian government-funded channel (along with iView like others). It responds to the same fix of deleting the fragment file ending .part-FragN and re-running the command to pick up from the last accepted part.
@jamestfans what happens if you run the yt-dlp command with
--downloader ffmpeg?This issue will soon be closed when #11972 is merged, but just as a note: I don’t think this problem manifests with abciview or zee5 downloads anymore anyways. They (along with olympics.com from #3578) all use akamai as their HLS CDN, and there seems to have been some sort of akamai-wide upgrade to their HLS streaming. There is no sign of
BYTERANGEin any of their m3u8 rendition playlists anymore.I was having the same issue issue with iView on Windows only. I used option –downlader ffmpeg and the download completed successfully.
I don’t have to use this option however on the Linux version of yt-dlp, only on Windows
I have a very different approach. I use yt-dlp as a part of a Linux bash script.
My command line look like this: yt-dlp -f “bv*,ba” --downloader ffmpeg --print title --no-simulate --write-info-json --progress -o “%(video_autonumber-1)05d.%(autonumber)01d” $url
I don’t use any config file
The reason, I use %(autonumber)01d instead of %(ext)s, is to avoid the audio overwriting the video, since dr.dk now use same ext for both.
Links to subs, are retrieved from the info json file, downloaded with wget and converted form .vtt to .srt. The hole shebang is combined with mkvmerge.
@0HAg0
Yes I’m running on version 6.1 from John’s static build
ffmpeg version N-68334-ga87a52ed0b-staticBut seems to be fixed by using the ffmpeg you pointed to, you just have a typo in the links
https://github.com/yt-dlp/FFmpeg-Builds
Thanks for your response
Not fixed in yt-dlp 2022.07.18.
EDIT: [debug] Command-line config: [‘-fhls-617’, ‘-v’, ‘https://iview.abc.net.au/video/ZW1386A004S00’] [debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8 [debug] yt-dlp version 2022.07.18 [135f05e] (zip) [debug] Python 3.10.4 (CPython 64bit) - Linux-5.18.10-76051810-generic-x86_64-with-glibc2.35 (glibc 2.35) [debug] Checking exe version: ffmpeg -bsfs [debug] Checking exe version: ffprobe -bsfs [debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4 [debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, pyxattr-0.7.2, secretstorage-3.3.1, sqlite3-2.6.0 [debug] Proxy map: {} [debug] Using fake IP 1.153.251.4 (AU) as X-Forwarded-For [debug] [abc.net.au:iview] Extracting URL: https://iview.abc.net.au/video/ZW1386A004S00 [abc.net.au:iview] ZW1386A004S00: Downloading JSON metadata [abc.net.au:iview] ZW1386A004S00: Downloading webpage [abc.net.au:iview] ZW1386A004S00: Downloading m3u8 information WARNING: [abc.net.au:iview] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U [debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id [info] ZW1386A004S00: Downloading 1 format(s): hls-617 [debug] Invoking hlsnative downloader on “https://iview-vod-hls.akamaized.net/prod/zw1/ZW1386A004S00MA1D1/20220201025049/hls/500000/hdntl=exp=1658632067~acl=%2F*~data=hdntl~hmac=4a8b2c00afb4b2a8997543173532567822874eadf115262d5d41324204e2e733/stream.m3u8?aka_me_session_id=AAAAAAAAAACDt9xiAAAAAA8Y3yvRdIJqTTCBAtM6fr3EheE8edX%2FjpYlRXMFbHOtUPZ75bg%2F8+XiIizBZ5rI5k1ftBZx%2FGdp&aka_media_format_type=hls” [hlsnative] Downloading m3u8 manifest [hlsnative] Total fragments: 354 [download] Destination: Series 9 Ep 4 Episode 4 [ZW1386A004S00].mp4 [download] 1.4% of ~261.28MiB at 130.86KiB/s ETA 27:20 (frag 4/354)[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of 10) … [download] 1.4% of ~261.28MiB at 29.71KiB/s ETA 51:11 (frag 5/354)ERROR: Data must be padded to 16 byte boundary in CBC mode Traceback (most recent call last): File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 1431, in wrapper return func(self, *args, **kwargs) File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 1521, in __extract_info return self.process_ie_result(ie_result, download, extra_info) File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 1578, in process_ie_result ie_result = self.process_video_result(ie_result, download=download) File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 2661, in process_video_result self.process_info(new_info) File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 3160, in process_info success, real_download = self.dl(temp_filename, info_dict) File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 2860, in dl return fd.download(name, new_info, subtitle) File “/usr/local/bin/yt-dlp/yt_dlp/downloader/common.py”, line 444, in download ret = self.real_download(filename, info_dict) File “/usr/local/bin/yt-dlp/yt_dlp/downloader/hls.py”, line 354, in real_download return self.download_and_append_fragments(ctx, fragments, info_dict) File “/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py”, line 526, in download_and_append_fragments decrypt_fragment(fragment, self._read_fragment(ctx)), fragment[‘frag_index’], ctx) File “/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py”, line 360, in decrypt_fragment return unpad_pkcs7(aes_cbc_decrypt_bytes(frag_content, decrypt_info[‘KEY’], iv)) File “/usr/local/bin/yt-dlp/yt_dlp/aes.py”, line 11, in aes_cbc_decrypt_bytes return Cryptodome_AES.new(key, Cryptodome_AES.MODE_CBC, iv).decrypt(data) File “/usr/lib/python3/dist-packages/Cryptodome/Cipher/_mode_cbc.py”, line 246, in decrypt raise ValueError(“Data must be padded to %d byte boundary in CBC mode” % self.block_size) ValueError: Data must be padded to 16 byte boundary in CBC mode
Rerunning command bombs on different fragment, suggesting not a problem with source file?
EDIT: @sofa-singer said above that “–downloader aria2c --hls-use-mpegts” worked for him. It doesn’t for me.
EDIT: Using “–downloader ffmpeg” seems to work reliably for me.
@gamer191, running with
--downloader ffmpegworked.And thank you for the tip on the updates. I took care of that.