diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000..68e9d81 --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,29 @@ +name: Documentation +on: + push: + branches: + - master + - main +permissions: + contents: read + pages: write + id-token: write +jobs: + deploy: + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + runs-on: ubuntu-latest + steps: + - uses: actions/configure-pages@v5 + - uses: actions/checkout@v5 + - uses: actions/setup-python@v5 + with: + python-version: 3.x + - run: pip install zensical + - run: zensical build --clean + - uses: actions/upload-pages-artifact@v4 + with: + path: site + - uses: actions/deploy-pages@v4 + id: deployment diff --git a/docs/demos_gallery.md b/docs/demos_gallery.md new file mode 100644 index 0000000..7ddedf2 --- /dev/null +++ b/docs/demos_gallery.md @@ -0,0 +1,53 @@ +# Demos Gallery + +Explore our robot demonstrations and visual content through this interactive carousel. + +
+
+
+ Franka Robot Demo +
Franka Robot - Following a moving target while base joint follows a sine curve
+
+
+ Kinova Robot Demo +
Kinova Robot - Simulated robot with continuous joints and nullspace control
+
+
+ IIWA Robot Demo +
IIWA Robot - Simulated robot demonstration
+
+
+ Franka Eight Pattern +
Franka Robot - Following target and being disturbed (contact)
+
+
+ Franka Nullspace Control +
Franka Robot - Null space control demonstration
+
+
+ Franka Teleoperation +
Teleoperation using Vicon tracking system (Speed x4)
+
+
+ CRISP Overview +
CRISP Framework Overview
+
+
+ Robot Collection +
Compatible Robot Platforms
+
+
+
+
+ + + + +
+
+ +
+

Navigation: Use arrow keys ← → or click the buttons to navigate through the gallery.

+
diff --git a/docs/misc/calibrate_gripper.md b/docs/deprecated/calibrate_gripper.md similarity index 100% rename from docs/misc/calibrate_gripper.md rename to docs/deprecated/calibrate_gripper.md diff --git a/docs/misc/controllers.md b/docs/deprecated/controllers.md similarity index 100% rename from docs/misc/controllers.md rename to docs/deprecated/controllers.md diff --git a/docs/misc/create_own_config.md b/docs/deprecated/create_own_config.md similarity index 100% rename from docs/misc/create_own_config.md rename to docs/deprecated/create_own_config.md diff --git a/docs/misc/demos.md b/docs/deprecated/demos.md similarity index 100% rename from docs/misc/demos.md rename to docs/deprecated/demos.md diff --git a/docs/misc/diagnostics.md b/docs/deprecated/diagnostics.md similarity index 100% rename from docs/misc/diagnostics.md rename to docs/deprecated/diagnostics.md diff --git a/docs/misc/multi_machine_setup.md b/docs/deprecated/multi_machine_setup.md similarity index 100% rename from docs/misc/multi_machine_setup.md rename to docs/deprecated/multi_machine_setup.md diff --git a/docs/misc/new_robot_setup.md b/docs/deprecated/new_robot_setup.md similarity index 100% rename from docs/misc/new_robot_setup.md rename to docs/deprecated/new_robot_setup.md diff --git a/docs/design_philosophy.md b/docs/design_philosophy.md new file mode 100644 index 0000000..4c7136f --- /dev/null +++ b/docs/design_philosophy.md @@ -0,0 +1,112 @@ +# Design Philosophy + +We follow a few design strategies in CRISP. +This page describes our choices along with their respective pros and cons. + +Our most important goal is to find a **meeting point between robotics and machine learning engineers.** +On the one hand, robotics engineers are already familiar with ROS2 and experienced in control and manipulation in general. +On the other hand, machine learning engineers who might not be familiar with these tools prefer to work with python-only environments. +We try to bring the strengths of both worlds in this project. + +## Reuse Existing ROS2 Infrastructure + +We decided to build CRISP on top of [ROS2](https://docs.ros.org/) for several reasons: + +**Pros:** + +- **Large community**: Many users and developers already familiar with tools for interacting with ROS2 systems (e.g., [rviz2](https://github.com/ros2/rviz), [rqt](https://docs.ros.org/en/rolling/Concepts/Intermediate/About-RQt.html), [ros2 CLI](https://docs.ros.org/en/rolling/Tutorials/Beginner-CLI-Tools.html)). +- **Existing ecosystem**: Well-tested packages for robot drivers, sensors, and visualization that we can leverage directly. The [ROS Index](https://index.ros.org/) lists thousands of available packages. +- **Simplified setup with pixi + robostack**: We use [pixi](https://pixi.sh/) with [robostack](https://robostack.github.io/) to provide a conda-like development environment. +This removes the traditional friction of ROS2 installation (no more sourcing setup.bash files or managing system dependencies) and makes it accessible to ML engineers accustomed to Python-centric workflows. + +**Cons:** + +- **"Framework jail"**: ROS2 imposes certain patterns (nodes, topics, executors) that may feel constraining for simple use cases. + Users must work within the ROS2 paradigm even for straightforward scripts. + +In all objects, we try to abstract away ROS2 details behind simple Python APIs: +```python +from crisp_py import Robot + +robot = Robot(...) # ROS2 node creation is hidden and spinned up internally to receive/send data +robot.wait_until_ready() # Waits for ROS2 topics to be alive +print(robot.end_effector_pose) # Direct access to data without dealing with ROS2 messages +``` + +## Configuration: YAML Files with Programmatic Freedom + +CRISP uses YAML configuration files to define robots, sensors, and control pipelines. +This provides a declarative way to set up common scenarios without writing code. + +**Pros:** + +- **Quick iteration**: Change parameters without modifying source code. + +**Cons:** + +- **Less flexibility**: Complex behaviors may be hard to express in YAML rather than code. + +However, we also provide full programmatic access. +Users can instantiate objects directly in Python and modify them freely when the YAML approach becomes limiting. +This is particularly useful for: + +- Rapid prototyping and debugging +- *Custom components* that don't fit the YAML schema + +Here is an example for the YAML config for an environment: +```yaml +gripper_mode: "absolute_continuous" + +robot_config: + robot_type: "franka" + time_to_home: 2.0 + publish_frequency: 50.0 + home_config: [0.0, 0.1, 0.0,-1.94, 0.0, 2.0, 0.8] + +gripper_config: + min_value: 0.0 + max_value: 0.4 + joint_state_topic: /gripper/joint_states + command_topic: /gripper/command + +camera_configs: + - camera_name: "primary" + camera_frame: "primary_link" + resolution: [256, 256] + camera_color_image_topic: "third_person_camera/image_raw" + camera_color_info_topic: "third_person_camera/camera_info" + - camera_name: "wrist" + camera_frame: "wrist_link" + resolution: [256, 256] + camera_color_image_topic: "wrist_camera/color/image_rect_raw" + camera_color_info_topic: "wrist_camera/color/camera_info" + +sensor_configs: + - sensor_type: "force_torque" + shape: [6,] + name: "ft_sensor" + data_topic: "external_wrench" +``` + +See the [example config files](https://github.com/utiasDSL/crisp_gym/tree/main/crisp_gym/config) or [here](https://github.com/utiasDSL/crisp_py/tree/main/crisp_py/config), and [how to define your own configs](getting_started_config.md) for more details. + +## Data Collection directly in LeRobot Format, no rosbags + +CRISP collects data directly in [LeRobot](https://github.com/huggingface/lerobot) format at a single fixed frequency, rather than saving [ROS bags](https://docs.ros.org/en/rolling/Tutorials/Beginner-CLI-Tools/Recording-And-Playing-Back-Data/Recording-And-Playing-Back-Data.html) for post-processing. + +The LeRobot format stores episodes as [HuggingFace datasets](https://huggingface.co/docs/datasets/), making it easy to share, version, and load data for training. Each episode contains synchronized observations (images, joint states) and actions at a consistent frequency. + +**Pros:** + +- **Minimizes the gap between teleoperation and policy deployment**: The data you collect is immediately ready for training without conversion steps. What you record is exactly what your policy will see. +- **Consistent timing**: A single frequency ensures synchronized observations and actions, avoiding timestamp alignment issues common with bag files. +- **Simpler pipeline**: No need to manage bag files, replay them, and transform to training formats. +- **Easy sharing**: Datasets can be pushed directly to [HuggingFace Hub](https://huggingface.co/datasets) for collaboration. + +**Cons:** + +- **Less data captured**: You only save what's needed at the target frequency, potentially losing high-frequency sensor data that might be useful for debugging or alternative analysis. +- **Less flexibility**: Post-hoc resampling or different observation combinations require re-collection. + +The recording is handled by the [RecordingManager](https://github.com/utiasDSL/crisp_gym/tree/main/crisp_gym/record) and here is an example script showing how the recording works [record_with_leader_follower](https://github.com/utiasDSL/crisp_gym/blob/main/crisp_gym/scripts/record_lerobot_format_leader_follower.py) +More details on recording can be found in the [getting started with the gym](getting_started_gym.md) documentation. diff --git a/docs/examples_camera.md b/docs/examples_camera.md new file mode 100644 index 0000000..39c0d09 --- /dev/null +++ b/docs/examples_camera.md @@ -0,0 +1,80 @@ +# Camera Examples + +### Cameras + +The cameras that we tested are: + +- Any usb camera or webcam using the [ROS2 usb-cam](https://github.com/ros-drivers/usb_cam) package (see [a pixi wrapper here](https://github.com/danielsanjosepro/pixi_usbcam_ros2)), +- [Real Sense](https://github.com/IntelRealSense/realsense-ros/tree/ros2-master) which gives amazing ROS2 support, +- and [Orbbec](https://github.com/orbbec/OrbbecSDK_ROS2). + +Check the [getting ros2 side ready guide](getting_started_controllers.md) to see some examples with cameras + +## Run a camera node + +We first start a camera node that publishes images to ROS2 topics. +Check the [getting ros2 side ready guide](getting_started_controllers.md) to see other examples with cameras. +If you use an usb camera / webcam, you can try [pixi_usbcam_ros2](https://github.com/danielsanjosepro/pixi_usbcam_ros2) and start it with: +```bash +pixi run -e jazzy usb_cam +``` + +## Access images from `crisp_py` + +Now in the broadcasted images can be accessed from `crisp_py`. +Using the Camera class from `crisp_py`: +```python +"""Simple example for a camera. It shows the camera feed in a matplotlib window.""" + +import matplotlib.pyplot as plt +from crisp_py.camera import CameraConfig, Camera + +camera_config = CameraConfig( + camera_name="primary", + camera_frame="primary_link", + resolution=[256, 256], + camera_color_image_topic="/image_raw", + camera_color_info_topic="/image_raw/camera_info", +) + +camera = Camera(config=camera_config, namespace="") +camera.wait_until_ready() + +# Display camera feed +plt.ion() +fig, ax = plt.subplots() +ax.axis("off") + +frame = camera.current_image +im = ax.imshow(frame) +while True: + im.set_data(camera.current_image) + plt.pause(1.0 / 30.0) +``` + +Or by defining a camera in a YAML configuration file: +```yaml +camera_name: "primary" +camera_frame: "primary_link" +resolution: [256, 256] +camera_color_image_topic: "/image_raw" +camera_color_info_topic: "/image_raw/camera_info" +``` + +Then load the configuration and use the `make_camera` factory method (assuming that you added the config path to the `CRISP_CONFIG_PATH` environment variable as described in the [configuration guide](getting_started_config.md)): + +```python +"""Example showing how to load camera configuration from a YAML file.""" +import cv2 +from crisp_py.camera import make_camera + +camera = make_camera("your_config_file_name") +camera.wait_until_ready() + +# Display one frame +plt.imshow(camera.current_image) +plt.axis("off") +plt.show() +``` + + diff --git a/docs/examples_full_environment.md b/docs/examples_full_environment.md new file mode 100644 index 0000000..e56a2ca --- /dev/null +++ b/docs/examples_full_environment.md @@ -0,0 +1,3 @@ +# Full Environment Example + +TODO diff --git a/docs/examples_gripper.md b/docs/examples_gripper.md new file mode 100644 index 0000000..9cf9c2a --- /dev/null +++ b/docs/examples_gripper.md @@ -0,0 +1,58 @@ +# Gripper Examples + +This page provides a simple example on how to use grippers with `crisp_py`. + +## Run a gripper node + +We first start a gripper node that + +- Publishes gripper the value of the gripper `JointState`, where the position value range should be known for further configuration, +- Listens to commands of type `Float32` with the same range as the joint state values to open/close the gripper, +- Optionally, a service to disable the torque of the gripper motors. + +Check the [getting ros2 side ready guide](getting_started_controllers.md) to see the examples with grippers. + +For... + +- ... the Franka Hand, a node is directly started with the FR3 or Panda pixi robots publishes the gripper state and listens to commands automatically. +- ... Dynamixel-based grippers, you can use [dynamixel_wrapper](https://github.com/danielsanjosepro/dynamixel_wrapper). See the README.md to calibrate and get the range of the gripper. + +## Access Gripper from `crisp_py` + +For simple binary control of the gripper (open/close), you can use the `Gripper` class from `crisp_py` or the `make_gripper` factory function if you defined the gripper in a YAML configuration file (see [config guide](getting_started_config.md) for more information). +```python +"""Simple example to control the gripper.""" +import time + +from crisp_py.gripper import make_gripper + +gripper = make_gripper("gripper_franka") +# or using GripperConfig directly: +gripper.config.max_delta = 0.15 +gripper.wait_until_ready() + +# Open and close the gripper +gripper.open() +time.sleep(2.0) +gripper.close() +time.sleep(2.0) +gripper.shutdown() +``` + +If using a disable torque service, you can enable/disable the torque of the gripper motors: +```python +from crisp_py.gripper import Gripper, GripperConfig + +config = GripperConfig( + min_value=1046.0, + max_value=2065.0, + joint_state_topic="TODO/joint_states", + command_topic="TODO/command", + enable_torque_service="TODO/set_torque", +) +gripper = Gripper(config=config) +gripper.wait_until_ready() + +gripper.enable_torque() +gripper.set_target(0.5) # Set to mid position +``` diff --git a/docs/examples_robot.md b/docs/examples_robot.md new file mode 100644 index 0000000..416cffb --- /dev/null +++ b/docs/examples_robot.md @@ -0,0 +1,166 @@ +# Robot Examples + +This page provides a simple example on how to use robots with `crisp_py`. + +## Run a robot node + +We first start a robot node that + +- Publishes robot the joint states to a topic as a `JointState`, usually with the name `joint_states`, +- (Optional) Publishes robot the current pose to a topic as a `PoseStamped`, usually with the name `current_pose` (if not, `crisp_py`'s `RobotConfig` can be configured to use TF frames), +- Listens to commands of type `PoseStamped` that are published to a topic, usually with the name `target_pose` (to control the end-effector in Cartesian space), +- Listens to commands of type `JointState` that are published to a topic, usually with the name `target_joint` (to control the null space of redundant robots or for joint control), +- Optionally, listen to `WrenchStamped` messages to apply forces/torques at the end-effector. + +Check the [getting ros2 side ready guide](getting_started_controllers.md) to see the examples with robots. + +We will use the Franka Robotics FR3 robot as an example, but other robots can be used similarly. +You can run the FR3 robot node (simulated or real) using the [pixi_franka_ros2](https://github.com/danielsanjosepro/pixi_franka_ros2): +```bash +pixi run -e jazzy franka # or franka-sim for simulation +``` + +## Access Robot from `crisp_py` + +The `Robot` class from `crisp_py` that serves as a mere interface to interact with the robot, with minimal logic. +- To easily retrieve the latest joint states and end-effector pose, +- To send target poses and joint states to the robot, +- To switch between different `ros2_controllers` +- To `home()` the robot to a predefined position. +- To linearly interpolate between two poses with `move_to()` if using a cartesian controller. + +What it does not do: +- Some kind of trajectory planning or advanced motion planning. +- Safety checks (e.g., joint limits, singularities, collisions, etc.) +- IK computations... + +Most of the low levels of control and safety is handled by the controller or should be handled by a user of this interface. + + +```python +"""Simple example to control the robot.""" +import time + +from crisp_py.robot import make_robot + +robot = make_robot("fr3") +robot.wait_until_ready() # make sure that all topics have been received + +# %% First home +robot.home() # will activate a joint trajectory controller and home the robot + +# %% Check state +print(robot.end_effector_pose) +print(robot.joint_values) + +# %% Activate cartesian impedance controller +robot.controller_switcher_client.switch_controller("cartesian_impedance_controller") +# Optionally, load custom parameters for the cartesian controller +robot.cartesian_controller_parameters_client.load_param_config( + file_path="..." +) + +# %% Move to a target pose +target_pose = robot.end_effector_pose +target_pose.position.z += 0.1 # Move up 10 cm + +robot.set_target(pose=target_pose) # This will directly send the target to the robot, use move_to for linear interpolation +``` + +A more advanced example to move the robot around using viser (requires `viser` dependencies): +```python +import time + +import numpy as np +import viser +from viser.extras import ViserUrdf +from scipy.spatial.transform import Rotation +from robot_descriptions.loaders.yourdfpy import load_robot_description + +from crisp_py.robot import make_robot +from crisp_py.utils.geometry import Pose + +robot = make_robot("fr3") # Change to your robot type +robot.wait_until_ready() + +robot.config.time_to_home = 2.0 +robot.home() +start_pose = robot.end_effector_pose + +robot.controller_switcher_client.switch_controller("cartesian_impedance_controller") + +server = viser.ViserServer() + +urdf = load_robot_description("panda_description") # Change to your robot description loader +viser_urdf = ViserUrdf( + server, + urdf_or_path=urdf, + load_meshes=True, + load_collision_meshes=False, + collision_mesh_color_override=(1.0, 0.0, 0.0, 0.5), +) + +with server.gui.add_folder("Visibility"): + show_meshes_cb = server.gui.add_checkbox( + "Show meshes", + viser_urdf.show_visual, + ) + show_collision_meshes_cb = server.gui.add_checkbox( + "Show collision meshes", viser_urdf.show_collision + ) + +@show_meshes_cb.on_update +def _(_): + viser_urdf.show_visual = show_meshes_cb.value + +@show_collision_meshes_cb.on_update +def _(_): + viser_urdf.show_collision = show_collision_meshes_cb.value + +config_with_gripper = np.array([*robot.joint_values, 0.0]) +viser_urdf.update_cfg(config_with_gripper) + +trimesh_scene = viser_urdf._urdf.scene or viser_urdf._urdf.collision_scene +server.scene.add_grid( + "/grid", + width=2, + height=2, + position=( + 0.0, + 0.0, + # Get the minimum z value of the trimesh scene. + trimesh_scene.bounds[0, 2] if trimesh_scene is not None else 0.0, + ), +) +# Add interactive transform controls for the end effector. +transform_handle = server.scene.add_transform_controls( + "/end_effector_target", + position=start_pose.position, + wxyz=start_pose.orientation.as_quat(scalar_first=True), + scale=0.3, + line_width=3.0, +) + +# Add callback for when the transform handle is moved. +@transform_handle.on_update +def update_robot_target(handle: viser.TransformControlsEvent) -> None: + rot = Rotation.from_quat(handle.target.wxyz, scalar_first=True) + pose = Pose(position=handle.target.position, orientation=rot) + robot.set_target(pose=pose) + + +while True: + config_with_gripper = np.array([*robot.joint_values, 0.0]) + viser_urdf.update_cfg(config_with_gripper) + time.sleep(0.01) +``` + +With this, you can control the robot as shown in the following video: + +
+
+ +
+
+ +Have fun controlling your robot! diff --git a/docs/examples_sensor.md b/docs/examples_sensor.md new file mode 100644 index 0000000..3902e7f --- /dev/null +++ b/docs/examples_sensor.md @@ -0,0 +1,19 @@ +### Sensors + +You can add further sensors (Force Torque Sensor, Tactile Sensor...) by adding a custom `Sensor` that subscribes to a topic. +`Sensor` is simply a class that gives read-only access to the data being published on a topic with a given message type. +This is particularly useful when adding further observation to the manipulator environment directly in the config: +```yaml +sensor_configs: + - sensor_type: "force_torque" + shape: [6,] + # buffer_size: 30 + name: "ft_sensor" + data_topic: "/external_wrench" + - sensor_type: "tactile" + shape: [...,] + name: "tactile_sensor" + data_topic: "/tactile_data" + - ... +``` +This sensors will directly be added to the observation space of the manipulator environment. diff --git a/docs/getting_started.md b/docs/getting_started.md index c668694..7bae29f 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -2,399 +2,15 @@ !!! Info If anything in the guide seems unclear to you, do not hesitate to open an issue or start discussion in our repositories. - Our goal is to demistify robotics, not make it harder. + Our goal is to demystify robotics, not make it harder. Here is an overview of the CRISP framework (please check our paper for details). -![Stack overview](media/crisp_overview.png#only-light) -![Stack overview](media/crisp_overview_dark.png#only-dark) + + - -- [ ] 1. The first part is the setup for the low-level [crisp_controllers](https://github.com/utiasDSL/crisp_controllers). +- [ ] 1. The first part is the setup the ROS2 side, with the low-level [crisp_controllers](https://github.com/utiasDSL/crisp_controllers) for the robots and other nodes for cameras, grippers and sensors. - [ ] 2. Then, you will try moving the robot using [CRISP_PY](https://github.com/utiasDSL/crisp_py) python interface. -- [ ] 3. Then, you can optionally include additional cameras and other sensors in your setup. -- [ ] 4. Finally, you can set up [CRISP_GYM](https://github.com/utiasDSL/crisp_gym) - the Gymnasium interface - and start policy deployment or teleoperation. +- [ ] 3. Finally, you can set up [CRISP_GYM](https://github.com/utiasDSL/crisp_gym) - the Gymnasium interface - and start policy deployment or teleoperation. ![Robots](media/robots_transparent.png) - -## 1. Getting the low-level C++ [CRISP](https://github.com/utiasDSL/crisp_controllers) controller ready - -The computer running the CRISP controller needs a real-time patch for the controller to run smoothly and safely. You can check out the [Franka Robotics guide on how to set up a RT-patch.](https://frankarobotics.github.io/docs/installation_linux.html#setting-up-the-real-time-kernel) -On newer Ubuntu versions, you can use [Ubuntu Pro](https://ubuntu.com/real-time) for an easy setup. - -Then, check if your robot is already included in one of our demos, check [how to run a demo](misc/demos.md) from our [demos repository](https://github.com/utiasDSL/crisp_controllers_demos). You can then follow the instructions there to start your robot(s) using a Docker container. Some of them offer the possibility to run the demos with simulated robots to test the setup. - -If your robot is not included in the demos that is not problem. Check out [How to set up a robot that is not available in the demos](misc/new_robot_setup.md). Once you get the controllers running, feel free to open a pull request on our repo to add it to the demos! We highly appreciate that! - -## 2. :snake: Use the python interface [CRISP_PY](https://github.com/utiasDSL/crisp_py) to control the robot - -### Installation - -!!! Note - If you want to use the gymnasium interface, CRISP_PY will be automatically installed in the gym. You can therefore check the installation of [CRISP_GYM](#4-getting-started-with-crisp_gym) directly. - However, this section still gives you an idea on how to use CRISP_PY with your robot. We do not recommend to skip it. - -To use `CRISP_PY`, we recommend using [pixi](https://pixi.sh/latest/), a modern conda-like package manager. -It can be used in combination with [robostack](https://robostack.github.io/) to easily install ROS2 in any machine. -There are a few ways to get you started: - -_... use in your already existing pixi project:_ - -To use `CRISP_PY` in an already existing pixi project, you need to make sure that `ROS2` is available. -Check the [pixi.toml](https://github.com/utiasDSL/crisp_py/blob/main/pixi.toml) of `CRISP_PY` to see how this looks like. -Then you can add `CRISP_PY` as a pypi package: -```bash -pixi add --pypi crisp-python -``` -or -```bash -uv add crisp-python -``` -or -```bash -pip install crisp-python -``` -Double-check that everything is working by running: - -```bash -python -c "import crisp_py" # (1)! -``` - -1. This should not log anything if everything is fine - -_... install from source:_ - -```bash -git clone https://github.com/utiasDSL/crisp_py -cd crisp_py -pixi install -pixi shell -e humble -python -c "import crisp_py" # (1)! -``` - -1. This should not log anything if everything is fine - -Now you can try to control the robot! Check out the [examples](https://github.com/utiasDSL/crisp_py/blob/main/examples) for inspiration. - -### Try it out with the robot - -Make sure that the demo container is running in the background, as we will need it to access the robot. -From now on, you can instantiate `Robot` objects to control the robot. - -??? example "Example robot usage:" - ```py - from crisp_py.robot import Robot - from crisp_py.robot_config import RobotConfig - - robot_config = RobotConfig(...) - robot = Robot(namespace="...", config=robot_config) # (1)! - robot.wait_until_ready() # (2)! - - print(robot.end_effector_pose) - - - robot.controller_switcher_client.switch_controller( - "cartesian_impedance_controller", # (4)! - ) - x, y, z = robot.end_effector_pose.position - robot.set_target(position=[x, y, z-0.1]) # (3)! - - robot.shutdown() - ``` - - 1. This will get information from the robot asynchronously - 2. Make sure that we get information from the robot before trying to set targets or reading the pose of the robot. - 3. Set target 10 cm downwards. Careful not to send poses that are too far away from the current one! - 4. This will request the controller manager to activate the cartesian impedance controller. You can use it with other controllers like the operational space controller! - -## 3. Adding cameras, grippers, and further sensors to CRISP_PY - -### Cameras - -To add a camera, you will need to run it in a separate container as well. -The cameras that we tested are: - -- Any usb camera or webcam using the [ROS2 usb-cam](https://github.com/ros-drivers/usb_cam) package, -- [Real Sense](https://github.com/IntelRealSense/realsense-ros/tree/ros2-master) which gives amazing ROS2 support, -- and [Orbbec](https://github.com/orbbec/OrbbecSDK_ROS2). - -Check the [demos](misc/demos.md) to see some examples with cameras - -??? example "Example camera usage:" - ```py - import cv2 - from crisp_py.camera import Camera, CameraConfig - - camera_config = CameraConfig( - camera_name="primary", - resolution=(256, 256), # (1)! - camera_color_image_topic="image_raw", # (2)! - camera_color_info_topic="camera_info", - ) - - camera = Camera(config=camera_config) # (3)! - camera.wait_until_ready() # (4)! - - cv2.imshow("Camera Image", camera.current_image) # (5)! - cv2.waitKey(0) - - ``` - - 1. You can define a custom resolution, independently of the resolution of the published image. - 2. Set here the topic of your custom camera name. crisp_py uses compressed images, so make sure that this topic is available as well. - 3. You can also pass `namespace="..."` to give the camera a namespace. This is required for a bimanual setup. - 4. Make sure that we received an image. This will fail with a timeout if the topic is wrong or the camera is not publishing. - 5. This will show you the latest received image! - -### Grippers - -For gripper control, you need to make sure that a ROS2 node is running that accepts commands through a topic and publishes the state of the gripper. -To use a: - -- Franka Hand, you just need to start the demo. An [adapter](https://github.com/utiasDSL/crisp_controllers_demos/blob/main/crisp_controllers_robot_demos/crisp_controllers_robot_demos/crisp_py_franka_hand_adapter.py) is already running to allow you to control the gripper this way, -- Dynamixel motor to control a gripper, we used the well-maintained [dynamixel_hardware_interface](https://github.com/ROBOTIS-GIT/dynamixel_hardware_interface) with a position controller for the gripper. - -??? example "Example gripper usage:" - You can use the gripper in `crisp_py` with: - ```py - from crisp_py.gripper import Gripper, GripperConfig - - # config = GripperConfig.from_yaml(path="...") (1) - config = GripperConfig( - min_value=0.0, - max_value=1.0, - command_topic="gripper_position_controller/commands", - joint_state_topic="joint_states", - ) # (2)! - gripper = Gripper(gripper_config=config) # (3)! - gripper.wait_until_ready() # (4)! - - print(gripper.value) - - gripper.open() - # gripper.close() - # gripper.set_target(0.5) - ``` - - 1. You can load the configs from a yaml file. If you calibrate the gripper manually (check the crisp_py docs for more information) you can select this way your custom calibration file. - 2. Set the range of allowed commands (min stands for fully closed, max to fully open) and the topics for the gripper. You can check the topics using `ros2 topic list` - 3. You can also pass `namespace="..."` to give the gripper a namespace. This is required for a bimanual setup. - 4. Make sure that we received a gripper value. This will fail with a timeout if the topic is wrong or the gripper is not publishing. - -### Sensors - -You can add further sensors (Force Torque Sensor, Tactile Sensor...) by adding a custom `Sensor` that subscribes to a topic. -Check the examples for more information. - - -## 4. Getting started with [CRISP_GYM](https://github.com/utiasDSL/crisp_gym) - -Similar to `CRISP_PY`, we recommend using `pixi` to install `CRISP_GYM`. - -```sh -git clone https://github.com/utiasDSL/crisp_gym -cd crisp_gym -``` -Now, you should set a few things before installing everything. -Create a file `scripts/set_env.sh` which will be sourced every time that you run a command in your environment. -The script will not be tracked by git. -In this script you need to add a environment variables: - -- `ROS_DOMAIN_ID` **(Required)**: which is used to define nodes that should be able to see each other. In our [demos](misc/demos.md) they are set to 100 as default. -- `CRISP_CONFIG_PATH` **(Optional)**: which should be the path to a config folder similar to [config path of CRISP_PY](https://github.com/utiasDSL/crisp_py/tree/main/config). - If this environment variable is unset, the default configurations will be used. - Check [how to create your own config](misc/create_own_config.md) guide for more information. - - -=== "crisp_gym >=2.0.0" - - ```sh title="scripts/set_env.sh" hl_lines="2" - export GIT_LFS_SKIP_SMUDGE=1 # (1)! - export ROS_DOMAIN_ID=100 - export CRISP_CONFIG_PATH=/path/to/config1/folder:/path/to/config2/folder # optional - ``` - - 1. Required for now to install LeRobot - - Finally check the config (if using one) - ```bash - pixi run python scripts/check_config.py - ``` - - -=== "crisp_gym < 2.0.0" - - ```sh title="scripts/set_env.sh" hl_lines="2" - export GIT_LFS_SKIP_SMUDGE=1 # (1)! - export ROS_DOMAIN_ID=100 - export CRISP_CONFIG_PATH=/path/to/config/folder # optional - ``` - - 1. Required for now to install LeRobot - -If you want to work in a multi-machine setup (e.g. policy runs in a different machine as controllers), then check [how to setup multi-machine in ROS2](misc/multi_machine_setup.md). - - -```sh -source scripts/configure.sh # (1)! -pixi install -pixi shell -e humble-lerobot -python -c "import crisp_gym" - -``` - -1. This will set some environment variable pre-installation as well as checking that you defined the previous script properly. - -You can also check that your configs are set up with: - -```sh -pixi shell crisp-check-config -``` - -If the previous steps worked, then you are good to go. - -### Teleoperation: Record data in [LeRobotFormat](https://github.com/huggingface/lerobot) - -You can record data in `LeRobotFormat` to train a policy directly in [LeRobot](https://github.com/huggingface/lerobot). -You will need to use teleoperation to record data and we highly recommend using a leader-follower setup to generate episodes. - -#### Leader-follower - -The leader can be controlled by a human operator and the follower will mimic its motion. -Checkout `scripts/leader_follower_teleop.py` to get an idea on how the code works. -For your specific setup you need to: - -- Define your own `TeleopRobotConfig`, check [`teleop_robot_config.py`](https://github.com/utiasDSL/crisp_gym/blob/main/crisp_gym/teleop/teleop_robot_config.py). -- Define your own `ManipulatorEnvConfig`, check [`manipulator_env_config.py`](https://github.com/utiasDSL/crisp_gym/blob/main/crisp_gym/manipulator_env_config.py). - -Then, to record data use: -```sh -pixi run -e humble-lerobot crisp-record-leader-follower \ - --repo-id / # (1)! -``` - -1. Add `--help` to check other parameters to pass to the record function. - -The script is interactive. It will first ask to choose the desired configuration files for the recording and then allow you to record episodes interactively. -There are two recording methods currently available: - -- `keyboard` (default): It allows you to record episodes using the keyboard with the keys - - __r__(ecord start/stop) an episode, - - __d__(elete episode) after recording a failed episode, - - __s__(ave episode) after recording a succesful episode, - - __q__(uit) after finishing. -- `ros`: It uses the topic `recording_state` to catch `String` ROS2 messages to follow the same recording workflow as the keyboard. - With this you can implement custom recording devices to control the recording workflow - - ??? example "Using the FR3 pilot buttons of Franka Robotics as a recording device" - In our lab, we use the buttons of the leader robot as a recording device with a for of the [franka-buttons](https://github.com/danielsanjosepro/franka_buttons_ros2/tree/main) repository. - The following script uses the circle, cross, check and up buttons as a record, delete, save and quit commands respectively (this is also part of the repository): - ```py - """Send recording commands for an episode recorder node to start, stop recording, save episodes and quit using the franka pulot buttons.""" - import rclpy - from rclpy.node import Node - - from franka_buttons_interfaces.msg import FrankaPilotButtonEvent - from std_msgs.msg import String - - - class ButtonToRecordMessage(Node): - """Node that subscribes to the button event and toggles the gripper when the circle button is pressed.""" - - def __init__(self) -> None: - super().__init__("button_to_record_message") - - self.create_subscription( - FrankaPilotButtonEvent, "franka_pilot_button_event", self.button_callback, 10 - ) - - self.publisher = self.create_publisher(String, "record_transition", 10) - - # Add a cooldown to avoid multiple toggles - self._last_toggle = self.get_clock().now() - self._cooldown = 0.5 - - self.get_logger().info("ButtonToRecordMessage node started.") - - def button_callback(self, msg: FrankaPilotButtonEvent): - """Callback function for the button event. - - If circle pressed, then pass the command to the gripper client to toggle the gripper. - """ - if (self.get_clock().now() - self._last_toggle).nanoseconds < self._cooldown * 1e9: - return - - if msg.pressed: - if msg.pressed[0] == "circle": - self.get_logger().info("Circle button pressed. Sending a record message.") - self.publisher.publish(String(data="record")) - if msg.pressed[0] == "check": - self.get_logger().info("Check button pressed. Sending a save episode message.") - self.publisher.publish(String(data="save")) - if msg.pressed[0] == "cross": - self.get_logger().info("Cross button pressed. Sending a delete episode message.") - self.publisher.publish(String(data="delete")) - if msg.pressed[0] == "up": - self.get_logger().info("UP button pressed. Sending a quit command message.") - self.publisher.publish(String(data="exit")) - - self._last_toggle = self.get_clock().now() - - - def main(): - rclpy.init() - node = ButtonToRecordMessage() - rclpy.spin(node) - rclpy.shutdown() - - - if __name__ == "__main__": - main() - ``` - -After this, you can visualize the episodes with rerun visualizer and LeRobot utils: -```sh -pixi run -e lerobot python -m lerobot.scripts.visualize_dataset \ - --repo-id / \ - --episode-index 0 -``` -...or use the [online tool for visualization](https://huggingface.co/spaces/lerobot/visualize_dataset). - -!!! warning - LeRobot is subject to frequent changes. This command might change in future versions. - -#### Other teleop setups - -You can add further teleop options to [`teleop/`](https://github.com/utiasDSL/crisp_gym/blob/main/crisp_gym/teleop) and create -a similar record script to [`scripts/record_lerobot_format_leader_follower.py`](https://github.com/utiasDSL/crisp_gym/blob/main/crisp_gym/scripts/record_lerobot_format_leader_follower.py) - -### Train a policy - -You can use LeRobot train scripts to train a policy simply by running: -```sh -pixi run -e lerobot python -m lerobot.scripts.lerobot-train \ - --dataset.repo_id=/ \ - --policy.type=diffusion \ - --policy.push_to_hub=false -``` - -!!! warning - LeRobot is subject to frequent changes. This command might change in future versions. - -They provide the latest implementations of most VLA. -Check [LeRobot](https://github.com/huggingface/lerobot) for more information. - -### Deploy policy - -After training with LeRobot, you can deploy the policy with: -```sh -pixi run -e humble-lerobot crisp-deploy-policy # (1)! -``` - -1. The script will interactively allow you to choose a model inside `outputs/train`. If you want to explicitly pass a path you can override it with `--path` - -!!! warning - LeRobot is subject to frequent changes. This command might change in future versions. - -Good job, now you can evaluate your model! - diff --git a/docs/getting_started_config.md b/docs/getting_started_config.md new file mode 100644 index 0000000..1bc25e9 --- /dev/null +++ b/docs/getting_started_config.md @@ -0,0 +1,54 @@ +# How to create your own config + +You can bring your own config to CRISP, so that you are able to create your own environments, teleoperation setup, controllers... + +1. First create a config folder and give it the following structure (you do not need to add all config folders): + +```bash + my_crisp_configs + ├── envs/ + │ ├── my_env1.yaml + │ └── my_env2.yaml + ├── recording/ + │ └── my_recording_manager.yaml + ├── policies/ + │ ├── my_lerobot_ditflow.yaml + │ └── my_lerobot_diffusion.yaml + ├── teleop/ + │ └── my_teleop_setup.yaml + ├── control/ + │ ├── my_osc_controller.yaml + │ ├── my_joint_controller.yaml + │ └── my_cartesian_impedance_controller.yaml + ├── grippers/ + │ ├── my_gripper_config.yaml + │ └── my_second_gripper_config.yaml + ├── robots/ + │ └── my_robot_config.yaml + ├── cameras/ + └── └── my_camera_config.yaml + +``` +2. Then add it to your `CRISP_CONFIG_PATH`, ideally directly in your environment's activate script: +```bash +export CRISP_CONFIG_PATH=/path/to/my_crisp_configs +``` +3. Check that the config works. + +```bash +crisp-check-config # Do this inside an environment with crisp_py or crisp_gym installed +``` + +This should output your config if it can be loaded properly + +--- + +Now you can use this to create your own environments: + +```python +from crisp_gym.manipulator_env import make_env + +env = make_env(env_type="my_env1", namespace="my_robot_namespace_if_required") +``` + +Also the record and deploy scripts should be able to find your config now and allow you to load it. diff --git a/docs/getting_started_controller_details.md b/docs/getting_started_controller_details.md new file mode 100644 index 0000000..e4a664b --- /dev/null +++ b/docs/getting_started_controller_details.md @@ -0,0 +1,80 @@ + + +!!! Info + Please check the [controller implementation and configuration](https://github.com/utiasDSL/crisp_controllers/tree/main/src) for more details on these extra terms and how to enable them. + +
+
+ +
+
+ +The low-level controllers are torque-based controllers that take a `target_pose` or `target_joint` as input and compute the required torques to move the robot to that target. + +## Joint control + +For joint control, we use a simple PD controller to compute the desired torques based on the error between the current and target joint positions and velocities. +The torque command is computed as: + +$$ \boldsymbol{\tau}_\text{cmd} = \mathbf{K}_p \mathbf{e} + \mathbf{K}_d \mathbf{\dot{e}}$$ + +where: + +- \( \boldsymbol{\tau} \) is the vector of joint torques to be applied, +- \( \mathbf{K}_p \) is the diagonal matrix of proportional gains, +- \( \mathbf{K}_d \) is the diagonal matrix of derivative gains, +- \( \mathbf{e} = \mathbf{q}_\text{target} - \mathbf{q}_\text{current} \) is the position error, +- \( \mathbf{\dot{e}} = \mathbf{\dot{q}}_\text{target} - \mathbf{\dot{q}}_\text{current} \) is the velocity error, usually only \(-\mathbf{\dot{q}}_\text{current}\) is considered, as the desired velocity is often not set. This term acts as a damping term to slow down the motion. + +## Cartesian control + +For Cartesian Impedance control, we imagine a virtual spring-damper system between the current end-effector pose and the desired target pose. +The desired torque command $\boldsymbol{\tau}_\text{cmd}$ is computed by first calculating the desired force/torque at the end-effector $\mathcal{F}_\text{desired}$ as + +$$ \boldsymbol{\tau}_\text{cmd} = \mathbf{J}^\top \mathcal{F}_\text{target} + \boldsymbol{\tau}_\text{nullspace}$$ + +where: + +- \( \mathbf{J} \) is the robot's geometric Jacobian matrix w.r.t. the base or the world frame (can be configured). +- \( \boldsymbol{\tau}_\text{nullspace} \) is an optional nullspace torque to regulate joint positions. + +### Desired end-effector force/torque +In our case the desired end-effector force/torque is computed using a PD controller in Cartesian space: + +$$ \mathcal{F}_\text{desired} = \mathbf{K}_p (\mathbf{X}_\text{target}\ominus\mathbf{X}_\text{current}) - \mathbf{K}_d \mathbf{J} \mathbf{\dot{q}}_\text{current} $$ + +where: + +- \( \mathbf{X}_\text{target}\ominus\mathbf{X}_\text{current} \) is the 6D pose error between the target and current end-effector poses, and its computation depends on whether we are representing motions w.r.t. the world frame or the base frame (can be configured). +- \( \mathbf{K}_p \) is the diagonal matrix of Cartesian proportional gains. +- \( \mathbf{K}_d \) is the diagonal matrix of Cartesian derivative gains. +- \( - \mathbf{J}\mathbf{\dot{q}} = - \mathcal{V} \) is the twist (linear and angular velocity) of the end-effector. In the controller it acts as a damping term. + +### Nullspace control + +The nullspace torque term allows us to regulate the joint positions while controlling the end-effector in Cartesian space. +In our implementation, it simply follows a PD control law to drive the joints towards a desired nullspace position: + +$$ \boldsymbol{\tau}_\text{nullspace} = \mathbf{N} ( \mathbf{K}_{p,\text{ns}} \mathbf{e}_\text{ns} - \mathbf{K}_{d,\text{ns}} \mathbf{\dot{q}} )$$ + +where: + +- \( \mathbf{N} \) is the nullspace projector. +- \( \mathbf{K}_{p,\text{ns}} \) is the diagonal matrix of nullspace proportional gains. +- \( \mathbf{K}_{d,\text{ns}} \) is the diagonal matrix of nullspace derivative gains. +- \( \mathbf{e}_\text{ns} = \mathbf{q}_\text{ns,desired} - \mathbf{q}_\text{current} \) is the nullspace position error. +- \( \mathbf{\dot{q}} \) is the current joint velocity. In this case, the desired nullspace velocity is assumed to be zero. + +The nullspace position can be set with `robot.set_target_joint(...)` when using the Cartesian controller. +It will publish a target joint position which is interpreted as the nullspace target. + + +## Safety and extras + +The actual torque commands sent to the robot are clamped to the allowed torque limits and torque rate limits defined in the config. +We also add extra terms that can be add/enabled to the controllers so the final torque command is: + +$$ \boldsymbol{\tau}_\text{final} = \text{safety}(\boldsymbol{\tau} + \boldsymbol{\tau}_\text{extra}) $$ + +where $\text{safety}(...)$ clamps the torques to the allowed limits and $\boldsymbol{\tau}_\text{extra}$ can include friction compensation, gravity compensation (if not already included by the robot hardware interface), coriolis compensation, and joint limit avoidance torques... + diff --git a/docs/getting_started_controllers.md b/docs/getting_started_controllers.md new file mode 100644 index 0000000..321db20 --- /dev/null +++ b/docs/getting_started_controllers.md @@ -0,0 +1,96 @@ +# Getting started with the ROS2 side + +On the ROS2 side, we want to: + +1. We want to run a low-level C++ [CRISP](https://github.com/utiasDSL/crisp_controllers) controller in your manipulator. + The computer running the CRISP controller might need a real-time patch for the controller to run smoothly and safely. + You can check out the [Franka Robotics guide on how to set up a RT-patch.](https://frankarobotics.github.io/docs/installation_linux.html#setting-up-the-real-time-kernel) + On newer Ubuntu versions, you can use [Ubuntu Pro](https://ubuntu.com/real-time) for an easy setup. +2. Start nodes for the cameras, the grippers and other sensors. + +For both we provide (1) pixi ready-to-use repositories that launch the nodes or (2) docker container demos of robots test that are ready to use as well as for cameras, grippers and other sensors. +If your robot, camera, gripper or sensor is not listed below, checkout [how to setup a new robot](new_robot_setup.md) and get inspired by other examples to create your own setup. + +## Start ROS2 nodes with pixi (recommended) + +To start ROS2 nodes for your robot, camera or sensor we recommend using `pixi` as a ready-to-use solution to start ROS2 nodes. +Here are some examples of repositories with `pixi` support for different hardware. + +### Manipulators + +Robots that we tested are: + +- **FR3** (real and simulated): https://github.com/danielsanjosepro/pixi_franka_ros2 +- **FER/Panda** (real): https://github.com/lvjonok/pixi_panda_ros2 +- **IIWA** (simulated): https://github.com/danielsanjosepro/pixi_iiwa_ros2 + +### Grippers + +Grippers that we tested are: + +- **Franka Hand**: a node is included in the [fr3 pixi](https://github.com/danielsanjosepro/pixi_franka_ros2) and [panda pixi](https://github.com/lvjonok/pixi_panda_ros2) repositories which is started automatically with the robot. If the Franka Hand is not connected, the node will crash silently. Checkout [this config](https://github.com/utiasDSL/crisp_py/blob/main/crisp_py/config/grippers/gripper_franka.yaml) for using it with `crisp_py`. +- Any **Dynamixel**-based gripper: https://github.com/danielsanjosepro/dynamixel_wrapper - check README.md +- **Robotiq 2F-85**: work in progress... + +### Cameras + +Cameras that we tested are: + +- Any **USB camera** (with usb_cam): https://github.com/danielsanjosepro/pixi_usbcam_ros2 +- **Real Sense**: https://github.com/danielsanjosepro/pixi_realsense_ros2 + +### Sensors + +Sensors that we tested are: + +- **Anyskin** tactile sensor: https://github.com/danielsanjosepro/anyskin_ros2 +- **Force/Torque** sensors: work in progress... + +## Start ROS2 nodes with docker containers + +We provide ready-to-use docker container demos for different manipulators and cameras (as an alternative to `pixi`). +New demos are welcome, in particular if tested with real hardware. +Some other manipulators that could be added to this list is [Duatic](https://github.com/Duatic/dynaarm_driver), [Universal Robots](https://github.com/UniversalRobots/Universal_Robots_ROS2_Driver) or other dual setups. + +### Available Demos + +| Robots | Franka Robotics FR3 | FR Dual FR3 | IIWA 14 | Kinova Gen3 | +| :--- | :---: | :---: | :---: | :---: | +| MuJoCo simulated Hardware | ✅ | ✅ | ✅ | ✅ | +| Real Hardware | ✅ | ✅ | ❔[^1] | ❔[^1] | + +[^1]: Untested, but effort interface available. + +We also have some examples with cameras. + +| Robots | Real Sense | Any Camera / Webcam | Orbecc | +| :--- | :---: | :---: | :---: | +| Camera demo | ✅ | ✅ | ✅[^2] | + +[^2]: Available container at: https://github.com/danielsanjosepro/orbecc_container_ros2 + +### How to + +Clone the repo +```bash +git clone git@github.com:utiasDSL/crisp_controllers_demos.git crisp_controllers_demos +cd crisp_controllers_demos +``` + +!!! WARNING + Do NOT use **Docker Desktop**. Just go for the normal Docker CLI. + +Start your robot or camera with: +```bash +docker compose up launch_xxx +``` + +Check the [docker-compose.yaml](https://github.com/utiasDSL/crisp_controllers_demos/blob/main/docker-compose.yaml) for the available launch_xxx options. + +!!! WARNING + If you work in different machines, you might want to consider using a different RMW. + To use a different middleware just pass an extra environment variable: + ```bash + RMW= docker compose up ... + ``` + diff --git a/docs/getting_started_gym.md b/docs/getting_started_gym.md new file mode 100644 index 0000000..3cc69d9 --- /dev/null +++ b/docs/getting_started_gym.md @@ -0,0 +1,192 @@ +# Get started with `crisp_gym` + +First, clone the repository: +```sh +git clone https://github.com/utiasDSL/crisp_gym +cd crisp_gym +``` +Now, you should set a few things before installing everything. +Create a file `scripts/set_env.sh` which will be sourced every time that you run a command in your environment. +The script will not be tracked by git. +In this script you need to add a environment variables: + +- `ROS_DOMAIN_ID` **(Required)**: which is used to define nodes that should be able to see each other. In our [ROS2 nodes](getting_started_controllers.md) they are set to 100 as default. +- `CRISP_CONFIG_PATH` **(Optional)**: which should be the path to a config folder similar to [config path of CRISP_PY](https://github.com/utiasDSL/crisp_py/tree/main/config) or [config path of CRISP_GYM](https://github.com/utiasDSL/crisp_gym/tree/main/crisp_gym/config) but with your own custom configurations. + If this environment variable is unset, the default configurations will be used. + Check [how to create your own config](getting_started_config.md) guide for more information. + + +```sh title="scripts/set_env.sh" hl_lines="3" +export GIT_LFS_SKIP_SMUDGE=1 # (1)! +export SVT_LOG=1 # (2)! +export ROS_DOMAIN_ID=100 +export CRISP_CONFIG_PATH=/path/to/config1/folder:/path/to/config2/folder # optional +``` + +1. This will avoid downloading large files when cloning the repository. You can always download them later with `git lfs pull`. +2. This will remove logging from SVT codecs, which are used to create data in LeRobot format. The logs can be quite verbose. + +If you want to work in a *multi-machine setup* (e.g. policy runs in a different machine as controllers/cameras), then check [how to setup multi-machine in ROS2](getting_started_multiple_machines.md). + +--- +Now we can install the environment: + +```sh +GIT_LFS_SKIP_SMUDGE=1 pixi install -e humble-lerobot +pixi shell -e humble-lerobot +python -c "import crisp_gym" +``` + +You can also check that your configs are set up with: + +```sh +pixi run -e humble-lerobot crisp-check-config +``` + +If the previous steps worked, then you are good to go. + +### Teleoperation: Record data in [LeRobotFormat](https://github.com/huggingface/lerobot) + +You can record data in `LeRobotFormat` to train a policy directly in [LeRobot](https://github.com/huggingface/lerobot). +You will need to use teleoperation to record data and we highly recommend using a leader-follower setup to generate episodes. + +#### Leader-follower + +The leader can be controlled by a human operator and the follower will mimic its motion. +Checkout `scripts/leader_follower_teleop.py` to get an idea on how the code works. +For your specific setup you need to: + +- Define your own `TeleopRobotConfig`, check [`teleop_robot_config.py`](https://github.com/utiasDSL/crisp_gym/blob/main/crisp_gym/teleop/teleop_robot_config.py). +- Define your own `ManipulatorEnvConfig`, check [`manipulator_env_config.py`](https://github.com/utiasDSL/crisp_gym/blob/main/crisp_gym/manipulator_env_config.py). + +Then, to record data use: +```sh +pixi run -e humble-lerobot crisp-record-leader-follower \ + --repo-id / # (1)! +``` + +1. Add `--help` to check other parameters to pass to the record function. + +The script is interactive. It will first ask to choose the desired configuration files for the recording and then allow you to record episodes interactively. +There are two recording methods currently available: + +- `keyboard` (default): It allows you to record episodes using the keyboard with the keys + - __r__(ecord start/stop) an episode, + - __d__(elete episode) after recording a failed episode, + - __s__(ave episode) after recording a successful episode, + - __q__(uit) after finishing. +- `ros`: It uses the topic `recording_state` to catch `String` ROS2 messages to follow the same recording workflow as the keyboard. + With this you can implement custom recording devices to control the recording workflow + + ??? example "Using the FR3 pilot buttons of Franka Robotics as a recording device" + In our lab, we use the buttons of the leader robot as a recording device with a fork of the [franka-buttons](https://github.com/danielsanjosepro/franka_buttons_ros2/tree/main) repository. + The following script uses the circle, cross, check and up buttons as a record, delete, save and quit commands respectively (this is also part of the repository): + ```py + """Send recording commands for an episode recorder node to start, stop recording, save episodes and quit using the franka pilot buttons.""" + import rclpy + from rclpy.node import Node + + from franka_buttons_interfaces.msg import FrankaPilotButtonEvent + from std_msgs.msg import String + + + class ButtonToRecordMessage(Node): + """Node that subscribes to the button event and toggles the gripper when the circle button is pressed.""" + + def __init__(self) -> None: + super().__init__("button_to_record_message") + + self.create_subscription( + FrankaPilotButtonEvent, "franka_pilot_button_event", self.button_callback, 10 + ) + + self.publisher = self.create_publisher(String, "record_transition", 10) + + # Add a cooldown to avoid multiple toggles + self._last_toggle = self.get_clock().now() + self._cooldown = 0.5 + + self.get_logger().info("ButtonToRecordMessage node started.") + + def button_callback(self, msg: FrankaPilotButtonEvent): + """Callback function for the button event. + + If circle pressed, then pass the command to the gripper client to toggle the gripper. + """ + if (self.get_clock().now() - self._last_toggle).nanoseconds < self._cooldown * 1e9: + return + + if msg.pressed: + if msg.pressed[0] == "circle": + self.get_logger().info("Circle button pressed. Sending a record message.") + self.publisher.publish(String(data="record")) + if msg.pressed[0] == "check": + self.get_logger().info("Check button pressed. Sending a save episode message.") + self.publisher.publish(String(data="save")) + if msg.pressed[0] == "cross": + self.get_logger().info("Cross button pressed. Sending a delete episode message.") + self.publisher.publish(String(data="delete")) + if msg.pressed[0] == "up": + self.get_logger().info("UP button pressed. Sending a quit command message.") + self.publisher.publish(String(data="exit")) + + self._last_toggle = self.get_clock().now() + + + def main(): + rclpy.init() + node = ButtonToRecordMessage() + rclpy.spin(node) + rclpy.shutdown() + + + if __name__ == "__main__": + main() + ``` + +After this, you can visualize the episodes with rerun visualizer and LeRobot utils: +```sh +pixi run -e lerobot python -m lerobot.scripts.visualize_dataset \ + --repo-id / \ + --episode-index 0 +``` +...or use the [online tool for visualization](https://huggingface.co/spaces/lerobot/visualize_dataset). + +!!! warning + LeRobot is subject to frequent changes. This command might change in future versions. + +#### Other teleop setups + +You can add further teleop options to [`teleop/`](https://github.com/utiasDSL/crisp_gym/blob/main/crisp_gym/teleop) and create +a similar record script to [`scripts/record_lerobot_format_leader_follower.py`](https://github.com/utiasDSL/crisp_gym/blob/main/crisp_gym/scripts/record_lerobot_format_leader_follower.py) + +### Train a policy + +You can use LeRobot train scripts to train a policy simply by running: +```sh +pixi run -e lerobot python -m lerobot.scripts.lerobot-train \ + --dataset.repo_id=/ \ + --policy.type=diffusion \ + --policy.num_inference_steps=10 +``` + +!!! warning + LeRobot is subject to frequent changes. This command might change in future versions. + +They provide the latest implementations of most major SOTA models in pytorch. +Check [LeRobot](https://github.com/huggingface/lerobot) for more information. + +### Deploy policy + +After training with LeRobot, you can deploy the policy with: +```sh +pixi run -e humble-lerobot crisp-deploy-policy --policy # (1)! +``` + +1. The script will interactively allow you to choose a model inside `outputs/train`. If you want to explicitly pass a path you can override it with `--path` + +!!! warning + LeRobot is subject to frequent changes. This command might change in future versions. + +Good job, now you can evaluate your model! + diff --git a/docs/getting_started_multiple_machines.md b/docs/getting_started_multiple_machines.md new file mode 100644 index 0000000..86de92f --- /dev/null +++ b/docs/getting_started_multiple_machines.md @@ -0,0 +1,111 @@ +To setup multiple machines, we recommend using a different RMW. +In our demos we provide the option to use Zenoh or CycloneDDS. +The setup with CycloneDDS uses multicast and might be a problem for your university/enterprise network. +You might want to try Zenoh in that case, which uses a router for node discovery. + +## Using CycloneDDS for multi-machine setups + +### In the host machine (where controllers run) + +To use the [demos repository](https://github.com/utiasDSL/crisp_controllers_demos) with `cyclone` as a middleware, pass the following environment variable to the services: +```bash +RMW=cyclone ROS_NETWORK_INTERFACE=enpXXXXXX docker compose up ... # (1)! +``` + +1. Modify this with your network interface: check `ip addr` on your shell. Otherwise it will just use `lo` as default. + +If you are using a custom robot or `pixi` installed robot, check the `setup_cyclone.sh` script to see how it is being configured. + +### In the remote machine (where the learning policy runs) + +1. Make sure that the Cyclone RMW is installed `ros-$ROS_DISTRO-rmw-cyclonedds-cpp`. If you use the `pixi.toml` provided + in this repo it should be the case. + + +2. Now you can modify your`scripts/set_env.sh` to include further configuration lines: + ```bash hl_lines="4-8" title="scripts/set_env.sh" + export ROS_DOMAIN_ID=100 # (1)! + export CRISP_CONFIG_PATH=/path/to/crisp_py/config # (1)! + + export ROS_NETWORK_INTERFACE=enpXXXXXX # (2)! + export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp + export CYCLONEDDS_URI=file:///path/to/crisp_gym/scripts/cyclone_config.xml # (3)! + + ros2 daemon stop && ros2 daemon start # (4)! + ``` + + 1. Check the [getting started](../getting_started.md) to see why we set this. + 2. Modify this with your network interface: check `ip addr` on your shell. + 3. __TODO__: this file as well as the path need to be modified! + 4. The communication daemon needs to be restarted to account for the changes. + +3. Finally, check that everything is working. +Enter in the humble shell with `pixi shell -e humble` and if your robot is active, run `ros2 topic list` and you should see some topics listed! + +## Using Zenoh for multi-machine setups + +> Zenoh /zeno/ is a pub/sub/query protocol unifying data in motion, data at rest and computations - from [Zenoh's website](https://zenoh.io/). + +[rmw_zenoh](https://github.com/ros2/rmw_zenoh) is a ROS middleware to use Zenoh as the pub/sub communication instead of DDS developed by Intrinsic. +To learn more about Zenoh, check their website and learn about the Zenoh middleware in their repository. + +What's important for us to know is that a router is required for discovery similar to how roscore worked in ROS1. +It can be configured with multicast and the avoid using the router but we will avoid a multicast setup since it might cause problems in some networks for example in universities. + +### In the host machine (where the controllers run) + +In the machine running the controllers, make sure that you start one Zenoh router. +In the [demos repository](https://github.com/utiasDSL/crisp_controllers_demos), we provide a service to start the router: +```bash +docker compose up launch_zenoh_router +``` + +This will start a Zenoh router, then all other nodes can be initialized. +To use the demos with `zenoh` as a middleware, pass the following environment variable to the services: +```bash +RMW=zenoh docker compose up ... +``` +The setup in the host machine is done! + +### In the remote machine (where the learning-based policy runs) + +In this part, we assume that you already installed the [CRISP_PY or CRISP_GYM](../getting_started.md#4-getting-started-with-crisp_gym). + +1. Make sure that the Zenoh RMW is installed `ros-$ROS_DISTRO-rmw-zenoh-cpp`. If you use the `pixi.toml` provided + in this repo it should be the case. + +2. Now you can modify your`scripts/set_env.sh` to include further configuration lines: + ```bash hl_lines="4-8" title="scripts/set_env.sh" + export ROS_DOMAIN_ID=100 # (1)! + export CRISP_CONFIG_PATH=/path/to/crisp_py/config # (1)! + + export RMW_IMPLEMENTATION=rmw_zenoh_cpp + export ZENOH_CONFIG_OVERRIDE='mode="client";\ + connect/endpoints=["tcp/YOUR_HOST_IP:7447"]' # (2)! + + ros2 daemon stop && ros2 daemon start # (3)! + ``` + + 1. Check the [getting started](../getting_started.md) to see why we set this. + 2. __TODO__: modify this to use the IP address + 3. The communication daemon needs to be restarted to account for the changes. + +3. Finally, check that everything is working. +Enter in the humble shell with `pixi shell -e humble` and if your robot is active, run `ros2 topic list` and you should see some topics listed! + +## Troubleshooting + +- Make sure that the Zenoh versions are the same across all the machines! +- Run `env | grep RMW_IMPLEMENTATION`, if the variable is not set, you need to make sure that the script `scripts/set_env.sh` is being executed! +- Be sure that the name of the interface is correct! + +## References + +- iRobot Middleware Config: [https://iroboteducation.github.io/create3_docs/setup/xml-config/](https://iroboteducation.github.io/create3_docs/setup/xml-config/) +- MoveitPro customize DDS: [https://docs.picknik.ai/how_to/configuration_tutorials/customize_dds_configuration/](https://docs.picknik.ai/how_to/configuration_tutorials/customize_dds_configuration/) + +- Cyclone Run-Time-Configuration: [https://github.com/eclipse-cyclonedds/cyclonedds/tree/a10ced3c81cc009e7176912190f710331a4d6caf#run-time-configuration](https://github.com/eclipse-cyclonedds/cyclonedds/tree/a10ced3c81cc009e7176912190f710331a4d6caf#run-time-configuration) +- StereoLabs improve performance: [https://www.stereolabs.com/docs/ros2/dds_and_network_tuning#change-dds-middleware](https://www.stereolabs.com/docs/ros2/dds_and_network_tuning#change-dds-middleware) +- Husarion DDS setup: [https://husarion.com/tutorials/other-tutorials/husarnet-cyclone-dds/](https://husarion.com/tutorials/other-tutorials/husarnet-cyclone-dds/) + +- ROS2 Doctor: [https://docs.ros.org/en/kilted/Tutorials/Beginner-Client-Libraries/Getting-Started-With-Ros2doctor.html](https://docs.ros.org/en/kilted/Tutorials/Beginner-Client-Libraries/Getting-Started-With-Ros2doctor.html) diff --git a/docs/getting_started_py.md b/docs/getting_started_py.md new file mode 100644 index 0000000..a416c99 --- /dev/null +++ b/docs/getting_started_py.md @@ -0,0 +1,50 @@ +# Get started with `crisp_py` + + +!!! Note + If you want to use the gymnasium interface, CRISP_PY will be automatically installed in the gym. You can therefore check the installation of [CRISP_GYM](#4-getting-started-with-crisp_gym) directly. + However, this section still gives you an idea on how to use CRISP_PY with your robot. We do not recommend to skip it. + +To use `CRISP_PY`, we recommend using [pixi](https://pixi.sh/latest/), a modern conda-like package manager. +It can be used in combination with [robostack](https://robostack.github.io/) to easily install ROS2 in any machine. +There are a few ways to get you started: + +_... use in your already existing pixi project:_ + +To use `CRISP_PY` in an already existing pixi project, you need to make sure that `ROS2` is available. +Check the [pixi.toml](https://github.com/utiasDSL/crisp_py/blob/main/pixi.toml) of `CRISP_PY` to see how this looks like. +Then you can add `CRISP_PY` as a pypi package: +```bash +pixi add --pypi crisp-python +``` +or +```bash +uv add crisp-python +``` +or +```bash +pip install crisp-python +``` +Double-check that everything is working by running: + +```bash +python -c "import crisp_py" # (1)! +``` + +1. This should not log anything if everything is fine + +_... install from source:_ + +```bash +git clone https://github.com/utiasDSL/crisp_py +cd crisp_py +pixi install +pixi shell -e humble +python -c "import crisp_py" # (1)! +``` + +1. This should not log anything if everything is fine + +Now you can try to control the robot! Check out the [examples](https://github.com/utiasDSL/crisp_py/blob/main/examples) for inspiration. +Or check the examples sections in the documentation [for the robot](examples_robot.md) or for other components such as [camera](examples_camera.md). + diff --git a/docs/index.md b/docs/index.md index c0758cd..62c5b0c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,13 +8,13 @@ hide: CRISP Controllers Logo + # CRISP - **C**ompliant **R**OS2 Controllers for Learn**i**ng-Ba**s**ed Manipulation **P**olicies *Authors: [Daniel San Jose Pro](https://danielsanjosepro.github.io)[^1], [Oliver Hausdörfer](https://oliver.hausdoerfer.de/)[^1], [Ralf Römer](https://ralfroemer99.github.io)[^1], Maximilian Dösch[^1], [Martin Schuck](https://amacati.github.io/) [^1] and Angela Schoellig [^1]*. - [^1]: The authors are with Technical University of Munich, Germany; TUM School of Computation, Information and Technology, Department of Computer Engineering, Learning Systems and Robotics Lab; Munich Institute of Robotics and Machine Intelligence. -> A collection of real-time, C++ controllers for compliant torque-based control for manipulators compatible with `ros2_control`. Developed for deploying high-level learning-based policies (VLA, Diffusion, ...) and teleoperation on your manipulator. It is compatible with any manipulator offering and effort interface. +> A collection of real-time, C++ controllers for compliant torque-based control for manipulators compatible with `ros2_control`. Developed for deploying high-level learning-based policies (VLA, Diffusion, ...) and teleoperation on your manipulator. It is compatible with any manipulator offering an effort interface. _If you use this work, please cite it using the [bibtex](#citing) below._ @@ -23,24 +23,71 @@ Check the [controllers (CRISP controllers) :simple-github:](https://github.com/u !!! info "Aloha gripper for Manipulators" Check out [aloha4franka](https://tum-lsy.github.io/aloha4franka/) for the gripper used in the videos. - -|