Gui Stencil Mask Optimization

[ZzzhHe]

Resolve #1538

Changes:

• Modified Presenter::render() to use three-pass rendering:
  1. First pass: Create stencil mask from GUI
  2. Second pass: Render world with stencil test (skip GUI-covered pixels)
  3. Third pass: Render GUI normally

[heinezen]

Hey thanks for contributing again :) Unfortunately, I think the solution you made clashes a bit with our design of the renderer.

In general, the renderer code is separated into two levels (see our renderer docs for details). The (low-level) level 1 interface handles the bare-metal stuff, e.g. interfacing with OpenGL to talk to the GPU, and should generally be inaccessible to other parts of the engine. Instead, the rest of the engine mainly interacts with the level 2 interface of the renderer which abstracts the low-level stuff away and implements systems that can be used for gameplay, e.g. playing sprite animations. The reason we have this level structure is mostly thread-safety, but also easier maintainance, since we can change or replace the level 1 implementaton without changing the entire engine. The latter is important for offering multiple renderer backends simultaneously to the user, e.g. OpenGL and Vulkan.

In your solution, there would be "bare-metal" OpenGL calls in the presenter, even though the presenter is technically part of the level 2 interface. If we do it like this, then the presenter does not work with a level 1 Vulkan renderer backend anymore, for example. Ideally, the presenter does not have to care what backend we initialize the renderer with, whether it's OpenGL or Vulkan.

[heinezen]

For solving this task properly, we should try to add stencil tests as a more general feature to the level 1 renderer first. If we want to activate it for the GUI, the stencil test write code should be moved into the GUI subsystem. To use the stencil tests in the other render passes, we could add settings to the RenderPass implementations; maybe stencil test reading could be a flag... When the render pass is rendered, we could switch on/off stencil testing based on that flag.

So in summary my thoughts would be:

1. Move the stencil test writing into the libopenage/renderer/gui subsystem of the level 1 renderer
2. Add settings to a RenderPass that allows us to configure activating/deactivating stencil tests (and maybe other settings in the future)
3. Based on the RenderPass settings, activate stencil testing when the render pass becomes active in

   openage/libopenage/renderer/opengl/renderer.cpp

   Line 170 in d6a4dd3

   void GlRenderer::render(const std::shared_ptr<RenderPass> &pass) {

[heinezen]

Apologies btw for askng for so many redesigns 😄 For issues that are not labelled "good first issue", the issue description can sometimes be a bit vague because we are ourselves not sure of what we want as a solution yet. You can always ask me for where to look in the docs before you do another issue like this one, maybe that helps :)

[ZzzhHe]

Thanks a lot for the detailed explanation! It all makes sense now. I’ll ensure to separate level 1 and level 2 as suggested and adjust my solution to follow this structure. I really appreciate your guidance!

[ZzzhHe]

Here are updates:

1. Added three stencil states in render pass to control rendering behavior:WRITE_STENCIL_MASK, USE_STENCIL_TEST, DISABLE_STENCIL.
2. Implemented stencil state handling in GlRenderer.
3. Added stencil_render_pass functionality in the GUI class.
4. Updated the Presenter class to handle Level 2 rendering abstractly, without OpenGL details.