yt-dlp: ffmpeg's `-movflags +faststart` fails on gvfs

Checklist

Region

France (maybe EU?)

Description

Trying to fetch https://www.france.tv/france-5/ernest-et-celestine/saison-1/3068333-pendant-que-l-orage-gronde.html with no particular CLI options, the downloaded file cannot be played with any of the players I have on my system. mpv notably reports

$ mpv Ernest\ et\ Célestine\ -\ Pendant\ que\ l\'orage\ gronde\ \[bc5bfd1c-2813-48d6-8c0e-524581bfce46\].mp4.old 
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: moov atom not found
[lavf] avformat_open_input() failed
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: moov atom not found
[lavf] avformat_open_input() failed
Failed to recognize file format.

Exiting... (Errors when loading file)

Reverting back to yt-dlp==2021.10.22 allows the downloaded file to be played properly.

I tried diffing the files (my MPEG4 skills are rusty) and it looks like some chucks are missing at the very end of the file.

-rw-r--r-- 1 remi users 182480138 25 janv. 12:57 'Ernest et Célestine - Pendant que l'\''orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].mp4'
-rw-r--r-- 1 remi users 182068891 25 janv. 12:57 'Ernest et Célestine - Pendant que l'\''orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].mp4.2022.1.21'

In fact, the files are identical up to their common size. So really, the newer version of yt-dlp omits the end of the file, which probably include the index bits (is that what it’s called? sorry if I get the terminology wrong)

$ dd bs=182068891 count=1 if=Ernest\ et\ Célestine\ -\ Pendant\ que\ l\'orage\ gronde\ \[bc5bfd1c-2813-48d6-8c0e-524581bfce46\].mp4 | sha256sum 
1+0 enregistrements lus
1+0 enregistrements écrits
182068891 octets (182 MB, 174 MiB) copiés, 3,60738 s, 50,5 MB/s
cbdf4aa2343c555311c14e8ccb13122944ec381fc8cc9ca9649d4e2b9d7cb6a7  -
$ dd bs=182068891 count=1 if=Ernest\ et\ Célestine\ -\ Pendant\ que\ l\'orage\ gronde\ \[bc5bfd1c-2813-48d6-8c0e-524581bfce46\].mp4.2022.1.21 | sha256sum 
1+0 enregistrements lus
1+0 enregistrements écrits
182068891 octets (182 MB, 174 MiB) copiés, 3,49349 s, 52,1 MB/s
cbdf4aa2343c555311c14e8ccb13122944ec381fc8cc9ca9649d4e2b9d7cb6a7  -

I’ll try to bisect later, that’s all I have for now.

Verbose log

$ yt-dlp -vU "https://www.france.tv/france-5/ernest-et-celestine/saison-1/3068333-pendant-que-l-orage-gronde.html"
[debug] Command-line config: ['-vU', 'https://www.france.tv/france-5/ernest-et-celestine/saison-1/3068333-pendant-que-l-orage-gronde.html']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.01.21 [f20d607b0]
[debug] Python version 3.9.10 (CPython 64bit) - Linux-5.12.6-gentoo-x86_64-Intel-R-_Core-TM-_i5-4670_CPU_@_3.40GHz-with-glibc2.34
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.01.21, Current version: 2022.01.21
yt-dlp is up to date (2022.01.21)
[debug] [FranceTVSite] Extracting URL: https://www.france.tv/france-5/ernest-et-celestine/saison-1/3068333-pendant-que-l-orage-gronde.html
[FranceTVSite] 3068333-pendant-que-l-orage-gronde: Downloading webpage
[debug] [FranceTV] Extracting URL: francetv:bc5bfd1c-2813-48d6-8c0e-524581bfce46
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading desktop video JSON
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading mobile video JSON
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading signed dash manifest URL
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading MPD manifest
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading signed hls manifest URL
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading 1 format(s): hls-2218+dash-audio_fre=96000
[debug] Invoking downloader on "https://cloudreplay.ftven.fr/11d11784efe16/1086049405_france-domtom_TA.ism/ZXhwPTE2NDM0MzI3MjV+YWNsPSUyZjExZDExNzg0ZWZlMTYlMmYxMDg2MDQ5NDA1X2ZyYW5jZS1kb210b21fVEEuaXNtKn5obWFjPWEwNDdiMzllZjZjM2IyZjJlY2ZjZTI0NzdkZjMyN2I2ZGZjZWI3ZmFhY2M2MzNmOGVjZjg3N2RiM2NjNDU4MjI=/1086049405_france-domtom_TA-video=2000000.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 91
[download] Destination: Ernest et Célestine - Pendant que l'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fhls-2218.mp4
[download] 100% of 171.06MiB in 00:36
[debug] Invoking downloader on "https://cloudreplay.ftven.fr/11d11784efe16/1086049405_france-domtom_TA.ism/manifest.mpd?hdnea=ip=78.196.20.108~exp=1643411725~acl=%2f11d11784efe16%2f1086049405_france-domtom_TA.ism*~hmac=faf3bffd25fe71d0f1e38101506550299a8c5c092ccbc44bf14a8db0f3b9e46f"
[dashsegments] Total fragments: 365
[download] Destination: Ernest et Célestine - Pendant que l'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fdash-audio_fre=96000.m4a
[download] 100% of 8.52MiB in 00:42
[Merger] Merging formats into "Ernest et Célestine - Pendant que l'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Ernest et Célestine - Pendant que l'"'"'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fhls-2218.mp4' -i 'file:Ernest et Célestine - Pendant que l'"'"'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fdash-audio_fre=96000.m4a' -c copy -map 0:v:0 -map 1:a:0 -movflags +faststart 'file:Ernest et Célestine - Pendant que l'"'"'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].temp.mp4'
Deleting original file Ernest et Célestine - Pendant que l'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fdash-audio_fre=96000.m4a (pass -k to keep)
Deleting original file Ernest et Célestine - Pendant que l'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fhls-2218.mp4 (pass -k to keep)

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Comments: 17 (5 by maintainers)

Most upvoted comments

Turns out I know the fella who wrote ffmpeg’s faststart so he pointed me to the underlying implementation. It’s fairly straightforward (once all the necessary checks are done): it opens a second file descriptor on the file and then cleverly reads a chunk from that second fd, buffers it and writes a previously read chunk on the original fd. The code that moves stuff around is here https://github.com/FFmpeg/FFmpeg/blob/d61240f8c95e9cf7a0aaef2bb4495960d3fec62c/libavformat/utils.c#L2038 for anyone interested.

In any case, this code works just fine on “normal” file systems and I’m very much inclined to lay the blame on gvfs’ sftp handling (which had indeed caused me some trouble in the past). I’ll just work around the issue by downloading locally and then copying the final file afterwards. Maybe I’ll poke Gnome/GLib people about this but this is probably a very niche feature.