Skip to content

Video in QuantumRender

Jump to bottom Edit New page
Kartikaya Gupta (kats) edited this page May 18, 2018 · 1 revision

This is a list of things that we are going to do for quantum video rendering support.

----------- Discussion from Nov 3rd (late) about passing texture data to web render (not only video)

rendering content looks like that:

  • [content process] FrameLayerBuilder generates the display list
    • ExternalImageDisplayItem created here with a certain ExternalImageKey
  • [content process] the display list is sent over IPC
  • [compositor thread] receives the transaction
  • [compositor thread] forwards the display list to WR's render backend thread
  • [compositor thread] do TextureHost related things (while the render backend is doing its work on the display lists)
    • match the ExternalImageKey with the corresponding TextureHost,
    • upload TextureHosts, present them to webrender using add_external_image(data, key)
  • [rende backend] compute the batches etc
  • [render backend] sends the final webrender::frame::Frame to the compositor
  • [compositor thread] webrender::render::Render submits the gl draw calls

If WR just use the gl context which is created by gecko, we might just send the gl texture to WR. <= is it always true that WR will use the gl context from gecko? If so, we can just pass the gl id to wr. otherwise, we can't do that.

problem:

  • if WR client and host are not in the same process, we will have the problem to pass the gl texture from client to host through WR ipc-channel
    • yes, jeff suggested serializing the displaylist in a shmem instead of a vector of bytes (what wr currently does), and pass the shmem through IPDL instead of rust IPC.
    • the IPDL message would be some kind of "WR Transaction" that contains the display list (as a shmem) and a list of PTextures associated to ExternalImageKeys
      • alternatively, we could just separate sending the display list (using rust ipc, talking directly to the render backend trhead) and the transaction (IPDL, talking to the compositor). but that sounds complicated and error-prone

With the new "WR Transaction", all WR components will at compositor process. Then we could create gl-texture from eglImage or shmem and pass the gl texture into WR

  • yes, although in some cases it could be useful to let webrender do the texture upload, for example:
    • it can do fancy asynchronous uploads that are more efficient than glTexImage2d
    • it can automatically atlas small things into the same texture (it already has the code to do that). which is not useful for video, but is useful for the small things we'll fallback on like native widgets and small svg elements
  • In any case, we need to support passing gl/d3d textures to WR directly, because some texture hosts (like DXGITextureHostD3D11 give you that, and not a raw buffer), so we can start with that, just need to make sure we don't block ourselves, so it is fair to start with that (especially since it sounds easier). => We will still have a interface to passing the "raw_buffer" from gecko to WR (for async texture uploading)
  • yes, but this interface does not need to be serialized like the display list, it's just a C function call happening on the compositor thread

Texture sharing plan:

  • passing the gl/d3d textures to WR directly (shmem will also upload to a certain of gpu texture at compositor thread)
    • to do this we could just use the existing code in BufferTextureHost, which creates a TextureSource (and does the upload in the process). The idea is to have a WebRenderCompositorOGL which implements the Compositor API (so that TextureHost can get the gl context and create the TextureSource), and the WebRenderCompositorOGL would do the glue between C++ stuff and webrender::Renderer (like call add_external_image).
  • handle gecko raw_buffer in WR. Then shmem could go though this path. sounds good to me!

---- YUV Image integration notes (nical, nov 4th):

  • Pull request (work in progress): https://github.com/servo/webrender/pull/528
  • adds a shader that works like the existing image shader and uses yuv conersion code copied from gecko's equivalent shader
  • adds a HashMap<ExternalImageKey, ExternalTexture> to the renderer
    • Most of the external image implementation is still up to you Jerry, I just added the minimum to be able to prototype out the rendering code
  • I am looking into ways to add non-external YUV images
    • a bit tricky due to the render backend expecting at most two source textures per primitive
    • also the resource cache doesn't seem to work well with the idea that a resource depends on several textures (invalidating one of them should invalidate the others)

--- Tuesday 8th

  • Importan next steps: get things to work on the compositor thread
    • Create WR Compositor (probably a fork of CompositorOGL) which holds the webrender::Renderer and the gl context
      • Make sure TextureHost/TextureSource work with this compositor just like it works with CompositorOGL
    • ask kats/jeff what the missing parts are for OMTC/e10s, and how we can help

nical:

  • YUV Shader (WIP)

  • Non-image-key YUV code path (WIP) This integrates with the texture cache etc.

  • some webrender internal refactorings to help with YUV stuff in general

  • Sotaro:

    • created a way to get imagehost textures to webrender (under the current non-compositor-thread situation)

  • Jerry:

    • ExternalImageKey work, callbacks api in webrender (no texture cache required here)

11/15 external image sequence:

  • Wr::render() is called
  • wr invokes the callback(externalId)
  • gecko finds the the ImageHost corresponding to the externalId
  • ImageHost finds the best texture ans assigns it to a compositableTextureHost/Source ref, calls Lock and PrepareTextureSource (just like gecko)
  • gecko returns the TextureId from the callback
  • Webrender does the rendering
  • after the frame is rendered, gecko goes through a list of all of the textureshosts that are locked, and unlock them

Next items:

  • Expose TextureClient/Host and CompositableClienr/Host to the webrender code paths and PWebRenderBridge
  • have a dedicated Webrender GL compositor(Stubbed implementation is in Bug 1312316)
  • how to do Directx11 handle -> gl texture id? Status sync
  • morris is working on Transaction-ify the webrender calls bug 1316903

Nical needs to ask toronto about:

  • who is working on the getting the integration branch <- kats
  • curious about the ANGLE vs D3D backend decision <- keeping the door open
  • what is the progress on displayitemification <- jeff-ish (scheduled for later, after hawaii)

=============== 11/22

  • IPC include order snafu
  • Progress on CompositorOGL
  • sotaro: working on getting reftests to work (using shmem for snapshot
  • sotaro: texture stuff
  • jerry: investigating black texture issue with external image
  • jerry: looking into using compositable client/host with WR
  • morris: adding prefs to disable/enable wr profiler

=============== 11/29

  • Discussions about PWebRenderBirdge/PTexture integration
  • synchronization issues between the render backend and the compositor (will need more discussions during the workweek)
  • let's do ImageClient/Host interation first
  • let's handle texture recycling separately
  • need to talk about scrollable layer during workweek
  • how to handle ClientLayerManager secific code all over the place
  • maybe try to abstract away some of the details

Jerry:

============== 12/20 Jerry:

TODO

  • Look into skipping building the frame if it hasn't changed (fast path when just swapping out textures)

sotaro:

  • ANGLE and android work

============

Jerry: trying to figure the wr update problem

Need to figure out how to split the CompositableHost between the compositor part and the renderer part, same for compositor, and texturehost/source (Jerry will be looking at this with ethan)

sotaro: working on making gl context dynamic

? what do we do with masks that are applied to compositables (hopefully let webrender do that)

blocksers: wr update render thread

Status update:

Jerry: dealing with compositable/texture lifetimes with render thread

Bind the external id with textureHost instead of compositorHost.

sotaro: progress on the dynamic gl context in webrender

TODO (video stuff)

  • experiment with creating a pipeline per video
  • external handle support
  • investigate each specific yuv format that we support and texture_external gl extension

3/14 Jerry: try to use macIOSurface with WR https://bugzilla.mozilla.org/show_bug.cgi?id=1347062

sotaro:

  • video rendering bug fixes
  • gleam update in wr (helps with enabling ANGLE)

3/21 Jerry: bug 1347062 - Use texture handle directly for video with webrender bug 1348684 - Handle external GL_TEXTURE_RECTANGLE handle in WR

Still debug for the black screen with MacIOSurface. The texture size of that macIOSurface texture in shader is (0, 0). So, maybe there are some wrong mac texture binding code in my patch. I try to get the macIOSurface texture content, but it returns all zero. mGL->fGetTexImage(LOCAL_GL_TEXTURE_RECTANGLE_ARB, 0, LOCAL_GL_RGB, LOCAL_GL_UNSIGNED_BYTE, buffer);

3/28 Jerry:

  1. yuv image interface https://github.com/servo/webrender/issues/1001
  2. DXGIYCbCrTextureHostD3D11 support with WR https://bugzilla.mozilla.org/show_bug.cgi?id=1351189

Sotaro:

  • ANGLE work
  • video integration

4/18 Jerry: NV12 supporting:

  1. Add a new image format RG8 in ImageFormat https://github.com/servo/webrender/pull/1126
  2. Add channel_index in ExternalImageHandler callback, ExternalImageData and UpdateForExternalBuffer https://github.com/servo/webrender/pull/1125
  3. Add GL_TEXTURE_EXTERNAL_OES texture target https://github.com/servo/webrender/pull/1132

4/25 Jerry:

  • WR NV12 support

https://github.com/servo/webrender/pull/1150 r?

Add a custom footer
Add a custom sidebar

Clone this wiki locally