[Bug]: Use gs.options.renderers.BatchRenderer failed

[YifeiNie]

Bug Description

Meet error when run

python examples/rigid/single_franka_batch_render.py

[ERROR]
Error at /project/src/render/vk/backend.cpp:137 in static LoaderLib *madrona::render::vk::LoaderLib::load()
Refetching vkGetInstanceProcAddr after dlsym failed

I'm sure renderer=gs.options.renderers.BatchRenderer leads to this problems, because I tested in my own env:

self.scene = gs.Scene(
    sim_options = gs.options.SimOptions(dt = self.dt, substeps = 1),
    viewer_options = gs.options.ViewerOptions(
        max_FPS = self.env_config.get("max_vis_FPS", 15),
        camera_pos = (-3.0, 0.0, 3.0),
        camera_lookat = (0.0, 0.0, 1.0),
        camera_fov = 40,
    ),
    vis_options = gs.options.VisOptions(
        show_world_frame = False,
        rendered_envs_idx = list(range(self.rendered_env_num)),
        env_separate_rigid = True,
        shadow = False,
    ),
    rigid_options = gs.options.RigidOptions(
        dt = self.dt,
        constraint_solver = gs.constraint_solver.Newton,
        enable_collision = True,
        enable_joint_limit = True,
    ),
    renderer=gs.options.renderers.BatchRenderer(
        use_rasterizer = True,
    ),
    show_viewer = self.env_config["show_viewer"],
)

If I comment renderer=gs.options.renderers.BatchRenderer(use_rasterizer = True,) it's ok or has the same error below.
May something in gs-madrona? I guess....
In /project/src/render/vk/backend.cpp around line 137, the code is:

#if defined(MADRONA_LINUX) || defined(MADRONA_MACOS)
    auto get_inst_addr = (PFN_vkGetInstanceProcAddr)dlsym(lib,
        "vkGetInstanceProcAddr");
    if (get_inst_addr == nullptr) {
        FATAL("Couldn't find vkGetInstanceProcAddr");
    }
    
    get_inst_addr = (PFN_vkGetInstanceProcAddr)get_inst_addr(
        VK_NULL_HANDLE, "vkGetInstanceProcAddr");
    if (get_inst_addr == VK_NULL_HANDLE) {
        FATAL("Refetching vkGetInstanceProcAddr after dlsym failed");
    }

    entry_fn = (void (*)())get_inst_addr;
#endif

'get_inst_addr' tries get itself by 'PFN_vkGetInstanceProcAddr', may this caused some problems?

Masters please help me

Steps to Reproduce

Run

python examples/rigid/single_franka_batch_render.py

with

• OS: [Ubuntu 20.04]
• GPU[RTX 3050]
• GPU-driver version (Driver Version: 570.133.07)
• CUDA / CUDA-toolkit version (12.4)
• torch.version.cuda (11.8)

Expected Behavior

Run python examples/rigid/single_franka_batch_render.py without error

Screenshots/Videos

No response

Relevant log output

Python 3.10.18
CUDA Toolkit 12.4
Pytorch 11.8

Error at /project/src/render/vk/backend.cpp:137 in static LoaderLib *madrona::render::vk::LoaderLib::load()
Refetching vkGetInstanceProcAddr after dlsym failed

Environment

• OS: [Ubuntu 20.04]
• GPU[RTX 3050]
• GPU-driver version (Driver Version: 570.133.07)
• CUDA / CUDA-toolkit version (12.4)
• torch.version.cuda (11.8)

Release version or Commit ID

aa797a9

SOLUTION

DONOT use pip to install gs-madrona, use source code, and comment codes as follows: in gs-madrona/src/render/vk/backend.cpp, recompile and reinstall

get_inst_addr = (PFN_vkGetInstanceProcAddr)get_inst_addr(
    VK_NULL_HANDLE, "vkGetInstanceProcAddr");
if (get_inst_addr == VK_NULL_HANDLE) {
    FATAL("Refetching vkGetInstanceProcAddr after dlsym failed");
}

[duburcqa]

@YifeiNie Thank you for your feedback. Sorry to hear gs-madrona is not working as expected on your machine. This rendering backend is still experimental for now, but hopefully it will be more stable in the future.

I will apply the patch you suggest (or rather avoid raising an exception in case of failure)

[duburcqa]

Cross-reference: Genesis-Embodied-AI/gs-madrona#37

[YifeiNie]

Hi @duburcqa, thanks for your reply.
I would like to reflect some other problems concern about batch renderer

No 'cam_znears_tensor' and 'cam_zfars_tensor' params

In Genesis/genesis/vis/batch_renderer.py, it uses MadronaBatchRendererAdapter in gs-madrona, and the input params are as follows:

self._renderer = MadronaBatchRendererAdapter(
    rigid, gpu_id, n_envs, n_cameras, n_lights, cameras_fov, *res, False, use_rasterizer
)

However, in gs_madrona/renderer_gs.py, the defination of the class is:

class MadronaBatchRendererAdapter:
    """Wraps Genesis Model around MadronaBatchRenderer."""

    def __init__(
        self,
        rigid,
        gpu_id,
        num_worlds,
        num_cameras,
        num_lights,
        cam_fovs_tensor,
        cam_znears_tensor,
        cam_zfars_tensor,
        batch_render_view_width=128,
        batch_render_view_height=128,
        add_cam_debug_geo=False,
        use_rasterizer=False,
    ):

where param cam_fovs_tensor and cam_zfars_tensor are lost, this will lead to core dump

torch.Tensor as data type

In gs_madrona/renderer_gs.py, line 82 and 83, there are:

cam_znear = cam_znears_tensor.cpu().numpy()
cam_zfar = cam_zfars_tensor.cpu().numpy()

Which means cam_fovs_tensor and cam_zfars_tensor are torch.Tensor type, but seems Pydantic does not support tensor type unless declare something at Genesis/genesis/options/renderers.py, line 104

class BatchRenderer(RendererOptions):
    """
    BatchRenderer renderer.

    Note
    ----
    This renderer is used to render the scene in a batch.

    Parameters
    ----------
    use_rasterizer : bool, optional
        Whether to use the rasterizer renderer. Defaults to False.
    """

    use_rasterizer: bool = False

and at Genesis/genesis/vis/batch_renderer.py, line 61

class BatchRenderer(RBC):
    """
    This class is used to manage batch rendering
    """

    def __init__(self, visualizer, renderer_options):
        self._visualizer = visualizer
        self._lights = gs.List()
        self._renderer_options = renderer_options
        self._renderer = None

        self._data_cache = {}
        self._t = -1

I believe a batched, fast and stable camera sensor will greatly help developers and researchers, looking forward to the stable version

[duburcqa]

@YifeiNie You are trying to align the main branch of gs-madrona with Genesis, which is incorrect. If you loo, more closely to Python dependencies requirements of Genesis in pyproject.toml, you will find out that the main branch of Genesis only support gs-madrona==0.0.3 currently.

[duburcqa]

Support for gs-madrona==0.0.4 is on the way.