yt-dlp: [abciview] [zee5] ERROR: Data must be padded to 16 byte boundary in CBC mode

Checklist

  • I’m reporting a bug unrelated to a specific site
  • I’ve verified that I’m running yt-dlp version 2022.05.18 (update instructions) or later (specify commit)
  • I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
  • I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
  • I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
  • I’ve read the guidelines for opening an issue

Description

Download begins then stops with this error: “ERROR: Data must be padded to 16 byte boundary in CBC mode”.

Verbose log

[debug] Invoking hlsnative downloader on "https://iview-vod-hls.akamaized.net/prod/zw3/ZW3040A001S00MA1D1/20210908051741/hls/4500000/hdntl=exp=1653131813~acl=%2f*~data=hdntl~hmac=968b90b5a9ec2efc1e020f0d280af19668f4d9fff35fcde9c17d1236543497c6/stream.m3u8?aka_me_session_id=AAAAAAAAAAAlyohiAAAAAIV7zCwd0HGHqUdzBIZg0ja5vozQgVmfzpL8R6FzBHOiCKK1yd4SaW0I3Lpa2721IdAe29m1IaW5&aka_media_format_type=hls"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 270
[download] Destination: /mnt/AU/Temporary/NA/20210924-ZW3040A001S00-Series 1 Ep 1.mp4
[download]   0.4% of ~975.58MiB at 559.83KiB/s ETA 29:37[download] Got server HTTP error: Downloaded 3784784 bytes, expected 3788768 bytes. Retrying (attempt 1 of 10) ...
[download]   0.4% of ~975.58MiB at 373.69KiB/s ETA 44:22 (frag 1/270)ERROR: Data must be padded to 16 byte boundary in CBC mode
Traceback (most recent call last):
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1395, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1479, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1535, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2640, in process_video_result
    self.process_info(new_info)
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3130, in process_info
    success, real_download = self.dl(temp_filename, info_dict)
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2838, in dl
    return fd.download(name, new_info, subtitle)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/common.py", line 445, in download
    ret = self.real_download(filename, info_dict)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/hls.py", line 348, in real_download
    return self.download_and_append_fragments(ctx, fragments, info_dict)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py", line 522, in download_and_append_fragments
    decrypt_fragment(fragment, self._read_fragment(ctx)), fragment['frag_index'], ctx)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py", line 365, in decrypt_fragment
    return unpad_pkcs7(aes_cbc_decrypt_bytes(frag_content, decrypt_info['KEY'], iv))
  File "/usr/local/bin/yt-dlp/yt_dlp/aes.py", line 10, in aes_cbc_decrypt_bytes
    return Cryptodome_AES.new(key, Cryptodome_AES.MODE_CBC, iv).decrypt(data)
  File "/home/pi/.local/lib/python3.9/site-packages/Cryptodome/Cipher/_mode_cbc.py", line 246, in decrypt
    raise ValueError("Data must be padded to %d byte boundary in CBC mode" % self.block_size)
ValueError: Data must be padded to 16 byte boundary in CBC mode

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Comments: 58 (13 by maintainers)

Commits related to this issue

Most upvoted comments

The author of #8081 identified this problem a while back:

I discovered this bug while trying to fix https://github.com/yt-dlp/yt-dlp/issues/3810, which I believe happens because Akamai sends invalid data when the Range header is used. With this PR, users can enable --no-continue as a workaround when downloading from affected sites. A proper fix for https://github.com/yt-dlp/yt-dlp/issues/3810 would be equivalent to enabling --no-continue by default when a site is known to use Akamai CDN.

(Note that using --no-continue is not a valid solution currently, it only would be with that PR’s changes)

IMO a better solution would be to make these changes:

  1. --no-continue should restart the download of the entire file, not just the fragment. It should also overwrite the temp file, not append to it
  2. incomplete fragments should never be resumed (which would fix this issue)

Hi all, just to advise I have spotted this issue on SBS, another Australian government-funded channel (along with iView like others). It responds to the same fix of deleting the fragment file ending .part-FragN and re-running the command to pick up from the last accepted part.

@jamestfans what happens if you run the yt-dlp command with --downloader ffmpeg?

This issue will soon be closed when #11972 is merged, but just as a note: I don’t think this problem manifests with abciview or zee5 downloads anymore anyways. They (along with olympics.com from #3578) all use akamai as their HLS CDN, and there seems to have been some sort of akamai-wide upgrade to their HLS streaming. There is no sign of BYTERANGE in any of their m3u8 rendition playlists anymore.

I found that it was giving this error using all the default settings - I played around and managed to get it to work flawlessly with the below adjustments. I can’t explain how/why it gets the error or how/why these adjustments fix it, but posting here in case it helps. --downloader aria2c --hls-use-mpegts

I was having the same issue issue with iView on Windows only. I used option –downlader ffmpeg and the download completed successfully.

I don’t have to use this option however on the Linux version of yt-dlp, only on Windows

I have a very different approach. I use yt-dlp as a part of a Linux bash script.

My command line look like this: yt-dlp -f “bv*,ba” --downloader ffmpeg --print title --no-simulate --write-info-json --progress -o “%(video_autonumber-1)05d.%(autonumber)01d” $url

I don’t use any config file

The reason, I use %(autonumber)01d instead of %(ext)s, is to avoid the audio overwriting the video, since dr.dk now use same ext for both.

Links to subs, are retrieved from the info json file, downloaded with wget and converted form .vtt to .srt. The hole shebang is combined with mkvmerge.

@0HAg0

Seems, you are using an different version of ffmpeg

Yes I’m running on version 6.1 from John’s static build

ffmpeg version N-68334-ga87a52ed0b-static

But seems to be fixed by using the ffmpeg you pointed to, you just have a typo in the links

https://github.com/yt-dlp/FFmpeg-Builds

Thanks for your response

Not fixed in yt-dlp 2022.07.18.

EDIT: [debug] Command-line config: [‘-fhls-617’, ‘-v’, ‘https://iview.abc.net.au/video/ZW1386A004S00’] [debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8 [debug] yt-dlp version 2022.07.18 [135f05e] (zip) [debug] Python 3.10.4 (CPython 64bit) - Linux-5.18.10-76051810-generic-x86_64-with-glibc2.35 (glibc 2.35) [debug] Checking exe version: ffmpeg -bsfs [debug] Checking exe version: ffprobe -bsfs [debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4 [debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, pyxattr-0.7.2, secretstorage-3.3.1, sqlite3-2.6.0 [debug] Proxy map: {} [debug] Using fake IP 1.153.251.4 (AU) as X-Forwarded-For [debug] [abc.net.au:iview] Extracting URL: https://iview.abc.net.au/video/ZW1386A004S00 [abc.net.au:iview] ZW1386A004S00: Downloading JSON metadata [abc.net.au:iview] ZW1386A004S00: Downloading webpage [abc.net.au:iview] ZW1386A004S00: Downloading m3u8 information WARNING: [abc.net.au:iview] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U [debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id [info] ZW1386A004S00: Downloading 1 format(s): hls-617 [debug] Invoking hlsnative downloader on “https://iview-vod-hls.akamaized.net/prod/zw1/ZW1386A004S00MA1D1/20220201025049/hls/500000/hdntl=exp=1658632067~acl=%2F*~data=hdntl~hmac=4a8b2c00afb4b2a8997543173532567822874eadf115262d5d41324204e2e733/stream.m3u8?aka_me_session_id=AAAAAAAAAACDt9xiAAAAAA8Y3yvRdIJqTTCBAtM6fr3EheE8edX%2FjpYlRXMFbHOtUPZ75bg%2F8+XiIizBZ5rI5k1ftBZx%2FGdp&aka_media_format_type=hls” [hlsnative] Downloading m3u8 manifest [hlsnative] Total fragments: 354 [download] Destination: Series 9 Ep 4 Episode 4 [ZW1386A004S00].mp4 [download] 1.4% of ~261.28MiB at 130.86KiB/s ETA 27:20 (frag 4/354)[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of 10) … [download] 1.4% of ~261.28MiB at 29.71KiB/s ETA 51:11 (frag 5/354)ERROR: Data must be padded to 16 byte boundary in CBC mode Traceback (most recent call last): File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 1431, in wrapper return func(self, *args, **kwargs) File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 1521, in __extract_info return self.process_ie_result(ie_result, download, extra_info) File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 1578, in process_ie_result ie_result = self.process_video_result(ie_result, download=download) File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 2661, in process_video_result self.process_info(new_info) File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 3160, in process_info success, real_download = self.dl(temp_filename, info_dict) File “/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py”, line 2860, in dl return fd.download(name, new_info, subtitle) File “/usr/local/bin/yt-dlp/yt_dlp/downloader/common.py”, line 444, in download ret = self.real_download(filename, info_dict) File “/usr/local/bin/yt-dlp/yt_dlp/downloader/hls.py”, line 354, in real_download return self.download_and_append_fragments(ctx, fragments, info_dict) File “/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py”, line 526, in download_and_append_fragments decrypt_fragment(fragment, self._read_fragment(ctx)), fragment[‘frag_index’], ctx) File “/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py”, line 360, in decrypt_fragment return unpad_pkcs7(aes_cbc_decrypt_bytes(frag_content, decrypt_info[‘KEY’], iv)) File “/usr/local/bin/yt-dlp/yt_dlp/aes.py”, line 11, in aes_cbc_decrypt_bytes return Cryptodome_AES.new(key, Cryptodome_AES.MODE_CBC, iv).decrypt(data) File “/usr/lib/python3/dist-packages/Cryptodome/Cipher/_mode_cbc.py”, line 246, in decrypt raise ValueError(“Data must be padded to %d byte boundary in CBC mode” % self.block_size) ValueError: Data must be padded to 16 byte boundary in CBC mode

Rerunning command bombs on different fragment, suggesting not a problem with source file?

EDIT: @sofa-singer said above that “–downloader aria2c --hls-use-mpegts” worked for him. It doesn’t for me.

EDIT: Using “–downloader ffmpeg” seems to work reliably for me.

@gamer191, running with --downloader ffmpeg worked.

And thank you for the tip on the updates. I took care of that.