Skip to content
This repository was archived by the owner on Sep 26, 2025. It is now read-only.

Enhance Docker workflow with unified mount system and improved cluster operations #56

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Enhance Docker workflow with unified mount system and improved cluster operations #56

wants to merge 10 commits into from

Conversation

Idate96
Copy link

@Idate96 Idate96 commented Jun 3, 2025

Summary

This PR introduces a unified mount configuration system for Docker containers and enhances cluster deployment workflows, making it easier to develop
with external IsaacLab/RSL-RL installations and deploy to HPC clusters.

Key Features

1. Unified Mount Configuration System

  • New mount-setup wizard for interactive configuration of external codebase mounts
  • Support for mounting external IsaacLab and RSL-RL installations into containers
  • Automatic generation of docker-compose.override.yaml from .mount.config
  • Commands: mount-enable, mount-disable, mount-set, mount-show

2. Enhanced Cluster Operations

  • Automatic mount synchronization: Local mount configurations are automatically applied on cluster
  • Two mount modes:
    • Sync mode: Automatically syncs local codebase to cluster
    • Mount-only mode: Uses existing codebase on cluster without syncing
  • Dynamic path expansion: Support for environment variables like $CLUSTER_USER in mount paths
  • Improved argument passing: Fixed script arguments being properly passed through SSH and job schedulers

3. Developer Experience Improvements

  • New run_dev.sh script for quick development container access
  • --rootless flag for environments without root Docker access
  • --fix-perms flag to automatically fix file ownership issues
  • Better error handling and debug output for troubleshooting

4. Documentation Updates

  • Comprehensive Docker setup guide with quick start section
  • Detailed cluster operations guide with mount configuration examples
  • Troubleshooting sections for common issues
  • Migration guide from previous container setups

Usage Examples

# Configure external mounts interactively
./docker/container.sh mount-setup

# Quick development access
./docker/run_dev.sh

# Run on university PC without root
./docker/run_dev.sh --rootless

# Deploy to cluster with external mounts
cd docker/cluster
./cluster_interface.sh push ext_template
./cluster_interface.sh job ext_template --task MyTask --num_envs 32000

Idate96 and others added 9 commits May 29, 2025 19:10
@Idate96 @claude
feat: Add improved Docker functionality and cluster workflow from mol…
3dd85b2 
…eworks_ext

- Replace simple Docker setup with comprehensive 2-container architecture
- Add production container (ext) for cluster deployment
- Add development container (ext-dev) with ROS2 and dual-mode support
- Implement unified mount system for optional IsaacLab/RSL-RL mounts
- Add cluster workflow scripts for SLURM/PBS job submission
- Include mount configuration system with validation
- Add container management script (container.sh) for easier usage
- Support both root and rootless Docker operation modes
- Add comprehensive documentation (MOUNT_SYSTEM_GUIDE.md, DOCKER_ARCHITECTURE.md)
- Update .gitignore to exclude user-specific configuration files

This brings the IsaacLabExtensionTemplate up to par with the advanced
Docker and cluster capabilities developed in moleworks_ext.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@Idate96
refactor: consolidate Docker documentation into README files
be95879 
- Merged DOCKER_ARCHITECTURE.md and MOUNT_SYSTEM_GUIDE.md into docker/README.md
- Created docker/cluster/README.md for cluster-specific operations
- Added missing sync_experiments.sh script from moleworks_ext
- Follows moleworks_ext documentation structure
@Idate96
fix: Apply rootless Docker permission fixes from moleworks_ext
72c8003 
- Remove extends directive from isaac-lab-ext-dev-rootless service
- Add full service definition with privileged: false for rootless mode
- Include all necessary service properties (network_mode, ipc, deploy, etc.)
- This ensures proper permissions without requiring privileged access
@Idate96
Improve Docker container UX and Python environment setup
a5e3832 
- Update Python aliases to use isaaclab.sh -p wrapper for proper environment setup
- Enhance container UX with 🐳 DOCKER prompt and welcome banner
- Fix bashrc sourcing to ensure proper shell environment loading
- Support Docker prompt indicator when home directory is mounted
- Add explicit entrypoint declarations in docker-compose services
@Idate96
Fix Docker entrypoint and remove figlet dependency
1c44bc8 
- Use dynamic_entrypoint.sh for development containers
- Remove figlet dependency that was causing startup errors
- Ensure proper Docker visual indicators are displayed
@Idate96
Fix Docker entrypoint path in docker-compose.yaml
cd58fdd 
- Correct entrypoint path from /dynamic_entrypoint.sh to /entrypoint.sh
- Ensure containers start properly with Docker visual indicators
@Idate96
Apply cluster fixes from moleworks_ext
4b9858d 
- Fix cluster user variable expansion in mount paths
- Fix script argument passing through SSH and job submission
- Add proper error handling for mount configuration parsing
- Add debug output for troubleshooting cluster jobs
@Idate96
Improve Docker and cluster documentation
856a65e 
- Add NVIDIA Container Toolkit prerequisite with installation link
- Clarify timestamped directory creation for cluster jobs
- Fix job output location references for SLURM/PBS
- Add troubleshooting for mount path variable expansion
- Add Docker/Apptainer version compatibility information
- Improve GPU troubleshooting with toolkit verification step
@Idate96
Remove RSL-RL installation and RTX sensor symlinks from Dockerfiles
090d70b 
- Remove automatic RSL-RL installation from both production and dev containers
- Remove RTX sensor symlink creation (no longer needed)
- Update documentation to reflect RSL-RL is not included by default
- Remove migration guide section from Docker README
@pascal-roth pascal-roth requested a review from Mayankm96 June 12, 2025 07:44
@Idate96
Enhance Docker system with comprehensive documentation and convenienc…
f6c0e6e 
…e script

## Key Improvements

### New Features
- Add missing run_dev.sh convenience script with user-friendly mode switching
- Add comprehensive environment file setup guide with examples
- Add system validation commands and performance expectations
- Add complete cluster workflow examples with timeline estimates

### Documentation Enhancements
- Add step-by-step environment configuration with generic placeholder examples
- Add performance benchmarks and system requirements based on testing
- Add troubleshooting for common setup issues (environment files, mount config, builds)
- Add job monitoring examples and cluster management workflows
- Add ETH Euler and generic university cluster configuration examples

### Validation & Testing
- Successfully tested complete Docker workflow from setup to cluster deployment
- Validated container building, job submission, and experiment sync
- Tested cluster integration with live job submission and monitoring
- All documentation examples verified through end-to-end testing

### User Experience Improvements
- Clear distinction between mandatory and optional configuration steps
- Quick start validation commands to verify setup before use
- Generic placeholders throughout (no personal details exposed)
- Enhanced troubleshooting based on real setup friction points discovered

The Docker system now provides a seamless workflow from local development to cluster deployment with comprehensive, tested documentation.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@Mayankm96 Mayankm96 Awaiting requested review from Mayankm96

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Idate96