streamlink: plugins.twitch: mid-roll ads ("Commercial Break in Progress") filtering issue with --twitch-disable-ads
Checklist
- This is a plugin issue and not a different kind of issue
- I have read the contribution guidelines
- I have checked the list of open and recently closed plugin issues
- I have checked the commit log of the master branch
Streamlink version
Latest stable release
Description
Basically the same as this issue, but its marked as resolved, so I can’t comment to say its back, apologies 🙏 #4106
Debug log
-
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 10
- Comments: 40 (21 by maintainers)
Bastimeyer, if you feel the need to apologize for your tone, then don’t be so brusque to begin with. You do not have to deal with anything. Not everyone possesses the same skillset as you, and berating people with a wall of text that doesn’t pertain to this issue is no more helpful than the spam you decry. Take a break from the Internet, if that’s what you need.
That said, ads are indeed not being filtered. Preroll ads do seem to be, but midroll ads are not. For whatever reason (I’m fairly technically savvy, but not a programmer), ads weren’t being displayed with Streamlink for quite some time without the need for
twitch-disable-ads. That flag currently blocks (most?) preroll ads, but not midroll.Nope, same happens without --twitch-low-latency flag. I just tested it. Also I never had any problems with midroll ads before today.
Just merged #4942, so that the main issue is fixed. The next nightly builds for users of Windows will be available in a couple of hours. Otherwise, install from the master branch via pip: https://streamlink.github.io/install.html#pypi-package-and-source-code
Streamlink 5.1.0 will be published soon.
I will also delete my secondary dev branch now, as it’s not needed anymore.
As said in #4942, I haven’t been able to find a mid-roll ad block again where the
--twitch-disable-adsad-filtering didn’t stop and the stream stayed paused forever. Not even sure if it’s an actual bug or if Twitch was simply serving infinite ads. If this keeps happening again, then please open a new issue and provide a proper log file, but do not open new issues if you don’t understand any of this. We’ve just added a new log level calledalland addedalllog messages to the HLS implementation’s M3U8 parser, so the same verbose logs can be generated as mentioned above, just with the--loglevel=allparameter instead of--loglevel=trace.--twitch-disable-adswas never meant to skip ads, this is just for filtering out embedded ads, as you can read in the docs.Skipping ads is an entirely different thing which we had also implemented several times in the past, and all of those implementations/workarounds had to be restored at some point because they caused other issues later on. This is done by applying certain HTTP headers or query string parameters to the API request / Twitch’s Usher service when acquiring a streaming access token for the HLS playlist. See the
--twitch-api-headeroption and why this specific CLI parameter was implemented. The git log / blame of the twitch plugin will also help you understand that.Prefetch segments get parsed regardless whether the option is set or not. The way prefetch segment handling is implemented could still cause issues with how the ads filtering works.
However, it’s also possible that Twitch annotates mid-roll ads differently, which incorrectly causes the last segment to be included in the output, which we don’t want.
The ads-filtering implementation works like this: Each HLS playlist response includes a set of tags for metadata and segments. Metadata includes date ranges, with annotated names, class names and other stuff. Ads are annotated this way, in specific date ranges. The stream segments included in the HLS playlist also have their own metadata, including program dates. Streamlink checks whether a segment’s program date is part of the annotated ad date range, and if that’s the case, then it starts filtering out the segments until the first one appears which doesn’t fall into the ad date range again.
In addition to that, segments each have a duration and optional title attribute. The segment titles are currently not being used when filtering out ads, even though they seem to contain specific names which are unlikely to be used by regular segments. For example, pre-roll ad segments at least contain the “Amazon” string in their title, which could be used as an additional indicator in addition to the date range checks. If there’s something wrong with the date ranges of mid-roll ads, then this could be used as a fallback.
However, I don’t know the contents of HLS playlists yet which contain mid-roll ads, so I could use some help.
Please apply the following diff and help debugging this, as we need to know the exact HLS playlist contents of the moment when a mid-roll ad gets embedded. This adds
tracelog messages to the log output for each line in the HLS playlist on each playlist refresh, and this will generate a ton of noisy output.Make sure to remove the existing log file before running Streamlink, or use a log file path with a dynamic name, like a timestamp.
Once you’ve found mid-roll ads, please upload the log file in its entirety to github gist and post the link. Be careful though, there might be personal information included in the HLS playlist contents.
I’m going to reopen this thread now and change its title.
PLEASE DO NOT COMMENT UNLESS YOU HAVE SOMETHING USEFUL TO SHARE WHICH WILL HELP FIXING THE ADS FILTERING (NOT SKIPPING/PREVENTING/ETC).
twitch-disable-adsjust waited for ads to end from what I can tell, its not able to actually skip them. So pauses the player/output until ads are over (as you saw for pre-roll start delay).From my testing without
twitch-low-latencyflag the stream/player will pause/freeze/wait until ads are over since it correctly detects both preroll/midroll ads are in progress. Withtwitch-low latencyflag, you wait for the player to start until preroll ads are done, and you get the ‘Commercial break in progress’ screen until midroll ads are over (to fix that requires what Bastimeyer described in the other thread to not see but would still pause/freeze your player on the last available frame)Currently working browser addons do the same with pausing playback until ads end, or replacing the stream with a 480p ad-free stream by proxy since certain regions don’t get particular ads (which seems out of streamlink’s scope?).
It’s likely due to this update below with an internal twitch change. https://discuss.dev.twitch.tv/t/an-updated-twitch-embedded-player-viewer-experience/41718 Embed type streams just didn’t used to get ads, and now they do. As mentioned in the other thread playerType=“embed” gets less/no ads so a new workaround or fix is needed now that that has been changed.
Wrong.
I have pushed a branch to my fork repo which you can test: https://github.com/streamlink/streamlink/compare/master...bastimeyer:streamlink:plugins/twitch/fix-prefetch-ads See the install docs on how to install Streamlink from different branches via pip: https://streamlink.github.io/install.html#pypi-package-and-source-code
I won’t be submitting a PR for this just yet, because I have not fully tested it, so this could also be complete garbage. Let’s see… The added test you can see in the diff which is built from the data I have gathered earlier today however fails without the modification of the custom
TwitchM3U8Parser, which is responsible for flagging segments as ads and which I hope I have fixed.A bit more as a clarification on why I think mid-roll ads didn’t get filtered out correctly: As explained, the prefetch segment implementation is not ideal. This is because of the lack of proper low-latency streaming support in the current version of the HLS protocol. Future versions of the protocol will/might/should fix this, so that streaming providers like Twitch don’t have to use custom solutions. Streamlink is therefore unfortunately relying on a kinda bad solution that shifts the live edge offset by copying the data from the last real (non-prefetch) segments unless a different solution gets used which adds more delay, and we don’t want that. What has been missed however was an extrapolation of the program date time of prefetch segments. This was not necessary in the past because ads were apparently not included in any prefetch data. Now prefetch segments can also be ads, and they are annotated by regular date-range tags. Since prefetch data does only contain a URL for fetching the data and since nothing else is included, what needs to be done when copying the data from the last regular segment is modifying the program date time of its copy and re-checking whether it is an ad. There’s also a second optimization. Regular segments can have an optional title. For normal streams, the title is “live”, and for VODs, the title is missing (I think, can’t remember). Ads however use a title with “Amazon” in it and an additional timestamp value. Looking for “Amazon” in the segment title should therefore also be done in addition to checking the date ranges.
Once I’m confident with my changes, I will open a PR for proper review. The branch is currently based on a different commit which I have opened a separate PR for, and this will need to be changed anyway. We also had already scheduled and then delayed the 5.1.0 release, which means a fix will definitely be included before we publish the next release.
Please test. Finding mid-roll ads is a bit tedious. And as already explained, this does only filter out ads, and even if it works properly without the “purple screen”, there will still be a stream discontinuity, which means bad players like old versions of VLC might still crash. This is a known issue which we can’t do anything about. As always, use recent software versions or better, switch to a different player like MPV if this keeps happening on recent versions of VLC as well. Enough with off-topic stuff now, though.
edit: There are still some issues with the mid-roll ad filtering. Just ran into an issue where it didn’t pick up the stream again. Unfortunately I didn’t do any trace logging. Annoying…
Thanks. I think I already have enough data with stuff mentioned in my last post. All I needed was a case where the filtering fails. My first run from the previous posts had ads filtered out correctly despite the prefetch segments, but my last one did not, so this should be enough for figuring out the issue. I will need to see how the cloning of the last regular segment affects the dateranges included for the prefetch data, where the ads already start. AFAIA this is not taken into consideration in the ad filtering implementation. I had already mentioned this in the last post of #4106.
You can keep running the trace logs, but it will most likely not be necessary anymore. Once I need new/more data, I will post an update.
I can confirm that I am also getting ‘commercial in progress’ when the streamer run ads manually mid-stream in order to avoid pre-rolls on stream. So while it looks like it works for pre-rolls it seems to fail when the streamer runs ads manually.
debug output for when the ads are run:
I’ll see if I can nail down whats happened, I’m a bit tech illiterate and use streamlink via a GUI. Ive watched ad-free for a few months now with Streamlink opening via Chatterino, which doesnt open the console window by default so I need to find out how I get a log. I tried adding --twitch-disable-ads to the options in case it were off (even though it seemed to be on before?) and now when the ad break happens, VLC just freezes and has to be force closed.
For context, this stream I watch every Wed (LevelUpYourGame) rolls commercial breaks between sets, so I know its ad time when the freezes occur. If it was on originally though, Im not sure if including --twitch-disable-ads twice may cause a crash itself.
Edit: Ive also gotten some prerolls on streams that dont run their own ad breaks today, same context otherwise.
This is issue #2198 due to the missing
#EXT-X-ENDLISTtag when streams go offline on Twitch. If you have set your timeout values too high, then Streamlink won’t terminate. Otherwise it terminates after 60 seconds due to the lack of data.However, it is possible that if the output is paused it is interfering with the timeout, because the timeout gets ignored during pauses. So if a stream goes offline during a pause, this is an issue. I didn’t run into this specific issue though.
But thanks for the hint, this needs to be fixed at some point. The solution I proposed in https://github.com/streamlink/streamlink/issues/2198#issuecomment-893524525 should be good enough.
No, that would be way too much and unnecessary noise. And storing cached message checks for only displaying a message once in a while is also not a great idea. We can improve the notes of the plugin file though which is shown in the plugins list, and we can also improve the description of the
--twitch-api-headersdescription, which gets shown in the docs,--helptext and manual.I’d like to ask for some more feedback.
As said, I am not 100% confident with the current state of my fixes which I have submitted in PR #4942, because I had two ad blocks over the past 24h now which caused the filtering to keep running, effectively pausing the stream forever. Other mid-roll ads were all filtered out fine, dozens of them. Unfortunately, I didn’t capture any logs during these two bad tests, so I don’t have any data for that and can’t reproduce the issue.
I have once again updated the branch I’ve commented about earlier (not the PR branch). The data I’m looking for is a full trace log without any truncation where mid-roll ads occurred and where the stream did not resume playing after the ads ended. Ad blocks can take several minutes now, so patience is needed. The branch doesn’t require any additional changes.
Once again, in order to install from my dev branch, use pip. The branch is this, so basically the commits of the PR and another one for temporary trace logging support: https://github.com/streamlink/streamlink/compare/master...bastimeyer:streamlink:plugins/twitch/fix-prefetch-ads Docs on how to install Streamlink from different branches via pip can be read here: https://streamlink.github.io/install.html#pypi-package-and-source-code
Then run
If you want to see the log while the stream is running, run
tail -f tracelog.txtfrom another shell instance.Please post and upload the compressed log here without removing any parts, especially not the log header, so that I know that you were actually running the right thing with the right arguments. If the log is too large and mid-rolls didn’t happen for a long time, then it’s probably better to use a different log and try again.
As said, I’m only looking for those cases where mid-roll ads caused the filtering to not end, even after 10 minutes or so. I need to know which data was included in the HLS playlist which caused this behavior.
Otherwise, please follow #4942 for any updates.
Thanks.
@zer0def
What do you mean with this?
I’m seeing
after it encountered live prefetch segments again after the ad break which started at
so everything looks like it’s okay. If you take a look at the playlist reload at this time where it resumes and compare the segment/prefetch URLs with the contents of the playlist before and after, this is all fine.
I’ve not been getting any mid-roll ads for the past one and a half hours or more. Not sure why. I’ve updated the branch that I’ve pushed earlier with the trace logging diff posted 8 hours ago, so it doesn’t have to be applied manually.
If you want to help getting a trace log output which can be used to diagnose the resuming issue (and potentially other issues as well which I didn’t catch), then please follow this after installing Streamlink from my dev branch as mentioned above, thanks: https://github.com/streamlink/streamlink/issues/4934#issuecomment-1310149207
edit: Had two mid-rolls since posting this, and those ads which were properly filtered out and the stream continued just fine. Still waiting for a mid-roll where the filtering incorrectly keeps going, as mentioned earlier. Didn’t make any code changes since then. So if anyone could help, that would be appreciated.
I’ve been getting this issue too, and I thought I’d point out easy twitch channels to test the ad-freezes on. Channels that stream Tarkov often manually play ads during the load-in phase before a game to prevent mid-rolls from interrupting the game. A big streamer called /WillerZ does this very reliably and is live right now. After 5 raids, the MPC-HC player froze when he manually ran an ad 100% of the time .
Thanks for working on this bug, and sorry if this isn’t helpful.
@Urbinholt This thread is not about ad prevention, it’s about ad filtering. Ad prevention is off-topic and you can read more about that here:
As a general notice: Please don’t comment on this thread unless you’ve got something valuable to contribute in order to fix the issue this thread is about. Every time new issues on Twitch occur, threads like this get very messy really fast, and we want to avoid this. Please understand and be patient.
Hi, i guess my issue is somehow related to this one. I’ve been using streamlink to fetch short (~8s) fragments of a stream, but today at midnight something happened and it broke functionality. It seems that twitch injected new preroll ad (it’s not even an ad actually, its splash screen saying “Preparing your stream…”) like one @donnydark0 posted. I’m using streamlink-cli like this:
streamlink -f -o output.ts https://www.twitch.tv/somestream audio_only --twitch-disable-hosting --twitch-api-header=Authentication="OAuth token" --twitch-disable-ads --hls-duration 8sAnd like that it return an error saying “Could not open stream <TwitchHLSStream> (No data returned from stream)” because ad is longer than 8 seconds and “twitch-disable-ads” actually doesn’t take it into account, so i have to manually adjust hls-duration length.Quick correction of my previous post. Prefetch segments were involved in the playlist request prior to the start of the filtering, and ad date ranges for that are included:
So I just had a mid-roll ad, and every single ad segment was properly filtered out by Streamlink. Prefetch segments were not included in the HLS playlist though, which means that prefetch segments are very likely the issue here, as described in #4106.
The only thing stream discontinuities cause, even if it’s a discontinuity in the same stream data and not just between the streams and an ad, are gaps in the program timestamps of the MPEG-TS bitstream, so players will handle this differently. MPV for example shows incorrect timings in the seek bar. Other players might still crash, eg. VLC, but if that’s the case, then there is literally nothing that Streamlink can do, and users will need to use a different player.
Here’s the playlist from the time when the first ad segments appeared
And here’s the continuation of the regular stream:
It’s completely irrelevant which application you’re using to launch Streamlink. This is not a Chatterino or other third party issue tracker. A debug log is required for any kind of plugin issue, as clearly stated in the issue template which you’ve completely ignored.
I have already shown you that ads get properly filtered out, if they occur, which they do now. The reason why you weren’t seeing any ads is that Twitch simply didn’t embed them into the stream. Simple as that. I have also already posted a link to the docs where you can read everything in detail, including why your version of VLC is crashing. If you’re having issues setting certain parameters from any third party frontends which launch Streamlink, then you can also set the options in one of Streamlink’s config files, which you can also read about in the documentation.
That’s all there is to say about that and any further discussion is unnecessary and will generate unnecessary noise.
Sorry for the harsh tone, but I am 100% sure that once I come back in a couple of hours, that this issue tracker, including the one of Streamlink Twitch GUI, as well as the discussion forum and Gitter/Matrix channels will be spammed by people who simply don’t read the docs or are otherwise completely unaware, and I will have to deal with all of that. Also on reddit already, which is an unofficial subreddit anyway, with comments from totally clueless users. Every time Twitch changes something, regardless whether Streamlink or Streamlink Twitch GUI is already prepared for it, stuff like this happens, and it steals my time, which is annoying. Read the documentation, especially if you’re calling yourself “tech illiterate”.