add LeRobot ACT notebook #3111

mhpanah · 2025-11-03T23:18:40Z

This PR adds a new tutorial notebook act_to_openvino.ipynb that demonstrates converting a Unitree Action Chunking Transformer (ACT) policy (from the LeRobot format) into OpenVINO IR (FP32), INT8. It walks through dependency verification, checkpoint preparation, ONNX export, OpenVINO Model Optimizer invocation, artifact validation, INT8 quantization and Evaluation.

Conversion Flow

Verify dependencies (torch, openvino, nncf, etc.).
Confirm presence of LeRobot
Load and reconstruct model (feature wrapping + config filtering).
Build ordered dummy inputs matching model signature.
Export ONNX
Run Model Optimizer to generate FP32 IR.
Confirm IR artifacts exist and report sizes.
Direct PyTorch to OpenVINO IR (No ONNX)
INT8 Quantization
Evaluation & Comparison Plotting

Core Dependencies:

openvino>=2025.0.0
torch>=2.1, torchvision, accelerate
nncf>=2.14.0 (future quantization integration)
safetensors, onnx, onnxruntime
Utility: numpy, pandas, matplotlib, tqdm, h5py, rich

review-notebook-app · 2025-11-03T23:18:46Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

openvino-dev-samples · 2025-11-04T00:40:38Z

notebooks/lerobot_act/lerobot-act.ipynb

@@ -0,0 +1,598 @@
+{


Line #1. # Export ACT Model to ONNX
Can we skip this step, and convert it to OV IR directly ?https://docs.openvino.ai/2025/openvino-workflow/model-preparation/convert-model-pytorch.html

Reply via ReviewNB

Added another section to also convert it to OV IR directly.

openvino-dev-samples · 2025-11-04T00:40:38Z

notebooks/lerobot_act/lerobot-act.ipynb

@@ -0,0 +1,598 @@
+{


I think we need an example to teach developer how to deploy this IR with OpenVINO

Reply via ReviewNB

Added evaluation section which shows how to run the model.

openvino-dev-samples · 2025-11-04T00:43:46Z

notebooks/lerobot_act/setup_unitree_lerobot_env.sh

+
+ENV_NAME="unitree_lerobot"
+LEROBOT_DIR="unitree_IL_lerobot"
+PY_VERSION="3.10"


Why should we limit the version of python and force developer to use Conda.
For user experience, a notebook should be run across different OS and python virtual environment.

It's due to LeRobot requirements.

sbalandi · 2025-11-06T12:44:52Z

notebooks/lerobot_act/setup_unitree_lerobot_env.sh

+  python -m pip install --upgrade pip wheel setuptools
+}
+
+install_openvino_requirements() {


please, add link to the quide https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide to the README instead of processing it here

sbalandi · 2025-11-06T13:24:12Z

notebooks/lerobot_act/lerobot-act.ipynb

@@ -0,0 +1,598 @@
+{


Line #38. def pip_install(*packages):
we have such function in utils, please, check example in Prerequisites here https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-native-agent-react/llm-native-agent-react.ipynb and use it

Reply via ReviewNB

sbalandi · 2025-11-06T13:24:12Z

notebooks/lerobot_act/lerobot-act.ipynb

@@ -0,0 +1,598 @@
+{


Where model.safetensors are originally placed ? On which step and from where should we download it ?

Reply via ReviewNB

We need to create a download link for our model or user can use their own model.

Could you, please, share some models so I can run it on my machine ?

sbalandi · 2025-11-06T13:24:12Z

notebooks/lerobot_act/lerobot-act.ipynb

@@ -0,0 +1,598 @@
+{


Is it possible to load the model directly into act_checkpoint ?

Reply via ReviewNB

sbalandi · 2025-11-06T13:24:12Z

notebooks/lerobot_act/lerobot-act.ipynb

@@ -0,0 +1,598 @@
+{


mo is deprecated, could you please convert model via ov.convert_model(model) https://docs.openvino.ai/2025/openvino-workflow/model-preparation/convert-model-pytorch.html ?

Reply via ReviewNB

…t act

openvino-dev-samples

Hi @mhpanah

I can not install the dependencies on my ubuntu system properly.
Have you verified everything from scratch on your side.

notebooks/lerobot_act/lerobot-act.ipynb

mhpanah · 2025-11-12T16:22:28Z

Hi @mhpanah

I can not install the dependencies on my ubuntu system properly. Have you verified everything from scratch on your side.

Please run this in your terminal before running the notebook:

bash setup_unitree_lerobot_env.sh
conda create -n unitree_lerobot python=3.10 -y
conda activate unitree_lerobot
# Launch notebook with correct kernel
jupyter lab act_to_openvino.ipynb --NotebookApp.kernel_name=unitree_lerobot

This is in the Minimal Installation & Launch section of the readme.

openvino-dev-samples · 2025-11-13T00:43:31Z

Hi @mhpanah
I can not install the dependencies on my ubuntu system properly. Have you verified everything from scratch on your side.

Please run this in your terminal before running the notebook:
bash setup_unitree_lerobot_env.sh
conda create -n unitree_lerobot python=3.10 -y
conda activate unitree_lerobot
# Launch notebook with correct kernel
jupyter lab act_to_openvino.ipynb --NotebookApp.kernel_name=unitree_lerobot
This is in the Minimal Installation & Launch section of the readme.

Thanks for clarification. However we can not limit using Conda only, and any specific dependencies of a notebook should be included in the .ipynb file instead of a separate script, which means you cannot expect users to install them through a terminal. Another approach maybe you can refer is that there is supplementary_materials folder to keep some special examples, which is more flexible for contributors. @sbalandi @aleksandr-mokrov whats your opinion ?

added lerobot act notebook

e83e59a

openvino-dev-samples reviewed Nov 4, 2025

View reviewed changes

aleksandr-mokrov requested a review from sbalandi November 4, 2025 12:52

sbalandi reviewed Nov 6, 2025

View reviewed changes

mhpanah added 2 commits November 7, 2025 16:37

added quantization, direct conversion and evaluation script to lerobo…

80df5e0

…t act

clearing cell outputs

ccaf1c6

mhpanah force-pushed the lerobot_act branch from bd88341 to ccaf1c6 Compare November 10, 2025 16:33

mhpanah requested review from openvino-dev-samples and sbalandi November 10, 2025 16:42

openvino-dev-samples reviewed Nov 11, 2025

View reviewed changes

notebooks/lerobot_act/lerobot-act.ipynb Show resolved Hide resolved

Renamed act_to_openvino.ipynb to lerobot-act.ipynb

765a7b1

mhpanah force-pushed the lerobot_act branch from 8503a81 to 765a7b1 Compare November 12, 2025 19:45

mhpanah requested a review from openvino-dev-samples November 12, 2025 19:47

add LeRobot ACT notebook #3111

Are you sure you want to change the base?

add LeRobot ACT notebook #3111

Conversation

mhpanah commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app bot commented Nov 3, 2025

Uh oh!

openvino-dev-samples Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhpanah Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

openvino-dev-samples Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhpanah Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

openvino-dev-samples Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

mhpanah Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

sbalandi Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

sbalandi Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbalandi Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhpanah Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

sbalandi Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

sbalandi Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbalandi Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openvino-dev-samples left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mhpanah commented Nov 12, 2025

Uh oh!

openvino-dev-samples commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mhpanah commented Nov 3, 2025 •

edited

Loading

openvino-dev-samples Nov 4, 2025 •

edited

Loading

openvino-dev-samples Nov 4, 2025 •

edited

Loading

sbalandi Nov 6, 2025 •

edited

Loading

sbalandi Nov 6, 2025 •

edited

Loading

sbalandi Nov 6, 2025 •

edited

Loading

sbalandi Nov 6, 2025 •

edited

Loading