360° video rendering

[tidoust]

It is already possible to render 360° videos within a <video> element using MSE and custom controls, but the Web application needs to handle control interactions and adjust the underlying adaptive streaming accordingly.

It is also possible to render 360° videos within a VR headset through the WebVR spec, and spatialize audio through the Web Audio API, but the application needs to work with bytes and pixels for that to happen.

In both cases, native support for 360° video rendering would both simplify the life of developers and, more importantly, allow to take advantage of the device's capabilities for improved performance and efficiency.

This was identified as a topic of interest in the report of the Web & Virtual Reality Workshop

[svgeesus]

@tidoust is there any recent activity on this topic?

[tidoust]

One idea in that space was to standardize projection and 3D information in media containers, which led to the Spherical Video proposal for ISO BMFF (MP4) and WebM. That does not seem to have moved in the past couple of years though.

W3C groups that look into 360° videos to some extent:

• The Immersive Captions Community Group is active and looks into "best practices for access, activation, and display settings for captions with different types of Immersive Media". This includes looking into captions for 360° videos.
• The Media & Entertainment Interest Group tracks 360° video support in 360 video support media-and-entertainment#27.
• The Immersive Web Community Group tracks 360° video support in Standardization of 360 video playback immersive-web/proposals#55. The group also discusses use cases for subtitles in 360° videos in Use case descriptions for subtitles in 360° immersive-web/proposals#39

In other words, there has not been recent activity on this topic that I'm aware of, but that is on the radar of different groups. From a Strategy team perspective, support for 360° video rendering has been raised again recently as a topic that could be worth looking into in explorations that @dontcallmedom and I are conducting to evaluate potential impacts of the metaverse on Web technologies. These explorations could perhaps lead to a workshop.