Incompatible SVC metadata

[aboba]

Since RTCEncodedVideoFrameMetadata does not inherit from WebCodecs EncodedVideoChunkMetadata, the SVC metadata is incompatible.

The EncodedChunkMetadata dictionary has structure to allow for future expansion of SvcOutputMetadata dictionary. The Expansion proposal is based on the information included within the Dependency Descriptor RTP header extension.

Issues with RTCEncodedVideoFrameMetadata compared with EncodedChunkMetadata:

1. Type mismatches:

unsigned long temporalLayerId, spatialLayerId vs.long spatialIndex, temporalIndex
unsigned short frameNumber vs. long long frameId
sequence <unsigned long> dependsOnIds vs. sequence <long long> dependencies

2. Missing information

sequence <unsigned long> decodeTargets

List of decode targets this frame participates in. Used to determine whether this frame should be forwarded to a receiver based on what decode targets the receiver is expecting.

map <unsigned long, unsigned long> chainLinks

Used to ensure we preserve decode order for the desired decode target. It is insufficient to satisfy the dependencies for the current frame.

[dontcallmedom-bot]

This issue was discussed in WebRTC WG 2023-04-18 – (Issue 170: Incompatible SVC Metadata (Bernard))

[alvestrand]

The proposal to add spatial index to WebCodecs is w3c/webcodecs#654

[aboba]

At TPAC joint meeting with the MEDIA WG, a new encoding API was described. Such an API, if implemented, would offer WebCodecs applications the ability to specify the reference buffers to be used in encoding a frame, enabling support for spatial (or quality) scalability as well as long term references. So we decided to hold off on adding spatialIndex to WebCodecs, since it might not be needed.