JS hooks for media codecs

[tidoust]

Browsers need to natively support image/audio/video codecs in order for content to be rendered. Polyfills are possible to work around lack of support but they require messing with the DOM e.g. to replace <img> tags.

A system enabling to write and distribute these algorithms in JavaScript and get them integrated in the overall media flow of user agents would provide much greater freedom to specialize and innovate in this space.

Performance would obviously be an issue although maybe not so much with the advent of WebAssembly, SIMD and Service Workers for background processing.

See the Custom image/audio/video codec APIs proposal in WICG.

[tidoust]

Partly discussed during the TPAC breakout session on Efficient media processing. Also somewhat linked to #130 although goal of the WebCodecs proposal is not to enable custom codecs per se.

[svgeesus]

Web Codecs seems to be addressing this; I note an issue on color space support

[tidoust]

The proposal here is not about exposing supported codecs to Web applications (WebCodecs indeed does that), but about enabling applications to register their own JS/WASM based codecs.

Applications can already provide their own media encoders/decoders in JS/WASM (often done in video conferencing tools for instance). They cannot register them so that e.g. their decoder would be used if they create a <video> tag that targets a media stream that is not natively supported by the browser (but that the custom decoder would support). I'm not aware of progress having been made in that space. It may be low priority in any case.