MGDP: Mastering a Generalized Depth Perception Model for Quadruped Locomotion

Project Website | Youtube Video

Abstract

Perception-based Deep Reinforcement Learning (DRL) controllers demonstrate impressive performance on challenging terrains. However, existing controllers still face core limitations, struggling to achieve both terrain generality and platform transferability, and are constrained by high computational overhead and sensitivity to sensor noise. To address these challenges fundamentally, we propose a generalized control framework: Mastering a Generalized Contrastive Depth Model (MGDP). We leverage NVIDIA Warp to enable efficient parallel computation of depth images, thereby mitigating the inherent high computational cost. MGDP extracts low-dimensional terrain feature representations from multi-modal inputs (depth images and height maps) and integrates an explicit depth map denoising mechanism. This process not only facilitates effective decoupling of perception from dynamics but also significantly reduces the memory. Furthermore, we design terrain-adaptive reward functions that modulate penalty strengths according to terrain characteristics, enabling the policy to acquire complex locomotion skills (e.g., climbing, jumping, crawling, squeezing) in a single training stage without relying on distillation. Experimental results demonstrate that MGDP not only endows the policy with superior cross-terrain generalization capability but also enables fast and efficient fine-tuning across diverse quadruped robot morphologies via its pre-trained, dynamics-decoupled perception model. This vigorously advances the development of unified, efficient, and generalized frameworks for quadrupedal locomotion control.

Installation

1. Create a Python virtual env with Python 3.8 (3.8.20 recommended).
   • conda create -n MGDP python=3.8.20
2. Install PyTorch 1.10 with CUDA 11.3:

   pip install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

3. Install Isaac Gym:
   • cd MGDP
   • cd isaacgym/python && pip install -e .
   • Try running an example: cd examples && python 1080_balls_of_solitude.py
4. Install this repo:
   • Clone the repository
   • pip install -e .
   • pip install -r requirement-gpu.txt
5. Install Warp sensors:
   • cd warp_sensor && pip install -e .
   • Test: warp-cam (exit: esc)

Usage

1. Stage 1: Train a Generalized Depth Perception Model (MGDP Stage 1)

   python legged_gym/scripts/train.py

   • Edit train.py to set args.task (e.g. random_dog_stage1), args.output_name, GPU id, etc.

2. Stage 2: Train a Generalized Perception-based Locomotion Controller (resume / fine-tune)

   python legged_gym/scripts/resume.py

   • Edit resume.py to set args.resume_name (previous run path), args.output_name (save path), and args.task (e.g. random_dog_stage2).
   • Select robot(s):
     • Set DOG_NAMES = [...] to mix multiple dogs in one run (envs use dog_id = i % len(DOG_NAMES)).
     • If DOG_NAMES is not set, it falls back to DOG_NAME.

3. Play / visualize

   • To visualize the Generalized Depth Perception Model:

     python legged_gym/scripts/vis_stage1.py

   • To visualize the Generalized Perception-based Locomotion Controller:

     python legged_gym/scripts/vis_stage2.py

4. View the terrain

   python legged_gym/scripts/play_terrain.py