Dynamic addition/removal of SourceBuffers

[dsilhavy]

This issue is a follow up on the discussion we had in the W3C Media and Entertainment IG calls. See also Additional Information below.

Overview

MSE based media players initialize the required SourceBuffer objects at the beginning of a media playback session. Typically, there will be one SourceBuffer for audio and one for video. After the initialization of the SourceBuffers they can not dynamically be removed and (re-)added again.

Description / Background / Use Cases

DASH multiperiod playback

In DASH based media streaming, the content is signaled in an MPD that can contain multiple periods. In a simple example, an MPD can consist of a main period, followed by an ad period followed by a main period again:

• Period 1: Main content with audio and video
• Period 2: Ad content with video and no audio
• Period 3: Main content with audio and video.

In the above example, Period 2 does not contain an AdaptationSet of type audio. With the current implementation of the MediaSource, playback will stall as soon as one of the attached SourceBuffers runs dry and has no data. For Period 2 there is no audio data available, consequently the SourceBuffer of type audio can only be filled with "silent dummy data".
An alternative to feeding "dummy data" into the SourceBuffer is to fully reinitialize die MediaSource and the SourceBuffers. This will have an impact on the user experience and pre-buffered video data is lost.

Media over QUIC

I am not actively following the work in MoQ but from my understanding one of the use cases might be related to this as well. In MoQ data can be prioritized, for instance it would be possible to assign a higher priority to the delivery of audio data than to the delivery of the video data in case of temporary network issues.

To allow playback to continue in this example, the video SourceBuffer should temporarily be removed and readded again once there is sufficient network capacity to receive video data again.

Suggested Solution

To account for the scenarios described above and allow applications to continue with a smooth playback if one of the initially created SourceBuffers can not be filled with data, we suggest allowing the dynamic removal and (re-)addition of SourceBuffers.

Additional Information

For related information, please refer to the meeting minutes of the W3C Media and Entertainment IG:

• Joint meeting W3C/SVTA
• MEIG Monthly Meeting - 2025/01/07

[dalecurtis]

Has just using multiple video elements been considered here?

[jyavenard]

How is this different than #360 ?

#357 would allow the usage scenario described here.

Re:

To allow playback to continue in this example, the video SourceBuffer should temporarily be removed and readded again once there is sufficient network capacity to receive video data again.

Say you're missing the audio, what does continue playback without audio provides over the current behaviour to pause (and element's readyState moves to HAVE_CURRENT_DATA)? and skipping back to live once data is fully received. you're missing crucial information.

same can be said about video.

[dsilhavy]

Thanks for the feedback, and apologies for the late answer

Has just using multiple video elements been considered here?

That is a potential solution but in my opinion there might be some side effects:

• Switching between video elements might take some time depending on the underlying platform
• Ad blockers might prevent applications from switching between video elements
• If the user is playing a video in fullscreen mode will a switch of the video element have a visible impact on the user
• The timing when to switch between the video elements might not be accurate

How is this different than #360 ?

In my opinion, #359 is a foundation for #360. I might want to create two SourceBuffers at the beginning (one for audio, one for video). During playback, the audio SourceBuffer is removed, and we continue playing video only.

Say you're missing the audio, what does continue playback without audio provides over the current behaviour to pause (and element's readyState moves to HAVE_CURRENT_DATA)? and skipping back to live once data is fully received. you're missing crucial information. same can be said about video.

I agree you might be missing crucial information but if your goal is to play at the bleeding live edge you might not want to resume playback from the position at which the potential stall occured. One demo that the MoQ group implemented was to show a still frame while audio still continued. Once there is video data available again playback continues at the bleeding live edge. From my understanding this is not possible today as playback can not continue unless we feed “dummy” data to the video SourceBuffer.

My personal opinion on this: If I am watching a sports livestream (e.g. a european football match) I prefer to hear audio at the live edge and look at a frozen frame instead of everything stalling and resuming at some point.

[chrisn]

My personal opinion on this: If I am watching a sports livestream (e.g. a european football match) I prefer to hear audio at the live edge and look at a frozen frame instead of everything stalling and resuming at some point.

This relates to the "playback through unbuffered ranges" use case raised in #160, which I agree is important.

[ntrrgc]

In streaming media sometimes there is a need to signal "there are no frames for this time period, do not consider this an underrun". For instance, in GStreamer this is called a gap event.

Signals like these are most notably used in sparse streams like subtitles, and they are normally emitted by the demuxers. Sinks are expected to treat gaps as silence or a still frame.

It could be possible to add an API like SourceBuffer.addGap() that receives a start and end timestamp. You can think of gaps as pseudo-frames. They would count for buffered ranges but no data is decoded for them.