shaka-player: DASH + ad insertion: Uncaught exception in promise in 4.7.0

I made a sample asset, and tested a basic solution along the lines of the second solution you mentioned.

I think there is just a fundamental problem in how I wrote #5950. The sample content I had, each period had a different codec but the same mimeType. And the solution I wrote handled that case, but it doesn’t handle variants that have multiple mimeTypes properly. Specifically, if there are multiple mimeTypes, we will test compatibility only with the first mimeType. For example, if we go from video/webm; codecs="vp09.00.41.08" to video/mp4; codecs="avc1.4d401f", we test to see if the stream is supported as though it was video/webm; codecs="vp09.00.41.08,avc1.4d401f"… which is wrong, because at no point are we using avc1 in webm, a combination not supported by Chrome.

This is kind of a problem because we still might create these variants, even though we can’t play them. Also because it means the documentation for manifest.dash.multiTypeVariantsAllowed is wrong.

I’ll make a change to fix how the multi-type support handles streams that switch between mimeTypes. MediaSource can play such content, we just get confused when we try to determine if they are playable or not. I think this will involve reverting how codecs are stored in streams (so they won’t be combined together anymore), and instead add a new value that just stores an array of the final full mimeTypes of each stream and uses that for compatibility checks.

I’m working with @littlespex and @dsparacio on this issue.

With our content, after the streams from all periods are combined, we end up with multiple video streams that include multiple codecs concatenated in the stream’s codecs property. For example: hev1.2.4.L63.90,hvc1.2.4.L63.90 and dvhe.05.01,avc1.42c00d.

Later on when the manifest is filtered for decoding configs and media capabilities, we hit the issue @littlespex described above. If codecs includes a comma, it’s assumed the stream is multiplexed (video + audio) and ultimately fails with HLS_COULD_NOT_GUESS_CODECS because the second codec is not a valid audio stream.

We identified this became an issue for us with the addition of the Handle mixed-codec variants feature: https://github.com/shaka-project/shaka-player/pull/5950, where codecs of the same content type are being concatenated.

https://github.com/shaka-project/shaka-player/blob/7fd99b7c2663903f3fa3991acfbab3782ddb0e07/lib/util/periods.js#L786-L795

I’m not sure I fully understand the intention here. Does a stream need to list both codecs for the mixed-codec feature to work properly? Are these codecs combined to say, “if the first codec is not available in a period but the second is, use the second”?

But then with https://github.com/shaka-project/shaka-player/pull/6047, filterDuplicateCodecs_ was added which essentially removes the second concatenated codec.

https://github.com/shaka-project/shaka-player/blob/4ae15c2c6fe2be98a7a6e9960ce48181f45e1324/lib/util/periods.js#L725-L760

That fix works when concatenated codecs have the same codec base, i.e. avc and avc avc1.42c01e, avc1.640028, but doesn’t work if the codec bases are different, i.e hev and hvc hev1.2.4.L63.90,hvc1.2.4.L63.90. Which probably explains why the originally reported issue was resolved with the fix but not with our content.

My outstanding questions regarding the approach we should take for a fix:

1. Should the concatenated codecs be filtered to remove the “duplicate” codec? We can add a fix here for the additional codecs that aren’t being caught. But does this break the mixed-codec feature?
2. Should the codecs remain concatenated? Then in the stream utils where multiplexed streams are split, add a check that only runs the multiplexed code if the concatenated codecs are different types (video vs audio).

I’m not sure I fully understand the intention here. Does a stream need to list both codecs for the mixed-codec feature to work properly? Are these codecs combined to say, “if the first codec is not available in a period but the second is, use the second”?

This was a fix for an issue from an internal client, who had multiperiod streams where some periods only had opus audio and some periods only had aac audio.

Should the concatenated codecs be filtered to remove the “duplicate” codec? We can add a fix here for the additional codecs that aren’t being caught. But does this break the mixed-codec feature?

I think it might break the mixed-codec feature, yeah. Annoyingly the sample content I was given for the original issue is no longer hosted online, so to confirm this I would have to make new content.

Should the codecs remain concatenated? Then in the stream utils where multiplexed streams are split, add a check that only runs the multiplexed code if the concatenated codecs are different types (video vs audio).

I think this would be safe? I’d feel happier if I could test it on some real content though.

Honestly, it’d probably be a good idea to make some content for reproducing the original issue that can be put into the Shaka Player demo anyway, so we can confirm that future changes don’t break this feature. I’ll hack something together tomorrow, and try those solutions out on it.