Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Unity plugin] Reference the tutorial in the Unity plugin docs #2249

Merged
merged 6 commits into from
Jan 19, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 32 additions & 14 deletions doc/unity.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,27 @@ Unity Plug-in
Introduction
------------

The MuJoCo `Unity plug-in <https://github.com/google-deepmind/mujoco/tree/main/unity>`_ allows the Unity Editor and
The MuJoCo `Unity plug-in <https://github.com/google-deepmind/mujoco/tree/main/unity>`__ allows the Unity Editor and
runtime to use the MuJoCo physics engine. Users can import MJCF files and edit the models in the Editor. The plug-in
relies on Unity for most aspects -- assets, game logic, simulation time -- but uses MuJoCo to determine how objects
move, giving the designer access to MuJoCo's full API.

An example project using MuJoCo's Unity plugin in a set of introductory tutorials are also available as a `standalone
repository <https://github.com/Balint-H/mj-unity-tutorial>`__.

.. _UInstallation:

Installation instructions
-------------------------

The plug-in directory (available at https://github.com/google-deepmind/mujoco/tree/main/unity) includes a
The `plug-in directory <https://github.com/google-deepmind/mujoco/tree/main/unity>`__ includes a
``package.json`` file. Unity's package manager recognizes this file and will import the plug-in's C# codebase to your
project. In addition, Unity also needs the native MuJoCo library, which can be found in the specific platform archive at
https://github.com/google-deepmind/mujoco/releases.
project. In addition, Unity also needs the native MuJoCo library, which can be found in the corrsponding `platform
archive <https://github.com/google-deepmind/mujoco/releases>`__. If you wish to simply use the plug-in and not
develop it, you should use one of the version-specific stable commits of the repository, identified by git tags. Check
out the relevant version of the cloned repository with git (``git checkout 3.X.Y`` where X and Y specify the engine
version). Simply using the ``main`` branch of the repository may not be compatible with the most recent release binary
of MuJoCo.

On Unity version 2020.2 and later, the Package Manager will look for the native library file and copy it to the package
directory when the package is imported. Alternatively, you can manually copy the native library to the package directory
Expand Down Expand Up @@ -115,7 +122,7 @@ This design principle has several implications:
- The layout of MuJoCo components in the GameObject hierarchy determines the layout of the resulting MuJoCo model.
Therefore, we adopt a design rule that **every game object must have at most one MuJoCo component**.
- We rely on Unity for spatial configuration, which requires vector components to be `swizzled
<https://en.wikipedia.org/wiki/Swizzling_(computer_graphics)>`_ since Unity uses left-handed frames with Y as the
<https://en.wikipedia.org/wiki/Swizzling_(computer_graphics)>`__ since Unity uses left-handed frames with Y as the
vertical axis, while MuJoCo uses right-handed frames with Z as the vertical axis.
- Unity transform scaling affects positions, orientations, and scale of the entire game object subtree. However, MuJoCo
doesn’t support collision of skewed cylinders and capsules (skewed spheres are supported via the ellipsoid primitive).
Expand Down Expand Up @@ -145,10 +152,6 @@ effects:
material assets for geom RGBA specification.
- It allows the importer to handle :ref:`\<include\> <include>` elements without replicating MuJoCo’s file-system
workflow.
- The current version of MuJoCo generates MJCF files with explicit :ref:`\<inertial\> <body-inertial>` elements, even when
the original model uses geoms for implicit definition of the body inertia. If you plan to change geom properties of
an imported model, remove these auto-generated ``MjInertial`` components manually. We plan to address this in a
future release of MuJoCo.

In Unity, there is no equivalent to MJCF’s “cascading” :ref:`\<default\> <default>` clauses. Therefore, components in
Unity reflect the corresponding elements’ state after applying all the relevant default classes, and the class structure
Expand Down Expand Up @@ -177,9 +180,10 @@ Scene recreation maintains continuity of physics and state in the following way:
persisted.
4. The MuJoCo state (for the joints that persisted) is set from the cache, and Unity transforms are synchronized.

Because the MuJoCo library doesn’t (yet) expose an API for scene editing, adding and removing MuJoCo components causes
complete scene recreation. This can be expensive for large models or if it happens frequently. We expect this
performance limitation to be lifted in future versions of MuJoCo.
MuJoCo has functionality for dynamic scene editing (through :ref:`mjSpec`), however, this is not yet
supported in the Unity plugin. Therefore, adding and removing MuJoCo components causes complete scene recreation. This
can be expensive for large models or if it happens frequently. We intend to lift this performance limitation to be in a
future versions of the plugin.

Global Settings
_______________
Expand Down Expand Up @@ -323,12 +327,26 @@ The plug-in allows using arbitrary Unity meshes for MuJoCo collision. At model
<http://www.qhull.org/>`__ to create a convex hull of the mesh, and uses that for collisions. Currently the computed
convex hull is not visible in Unity, but we intend to expose it in future versions.

Height fields
_____________

MuJoCo hfields are represented in Unity through terrain gameobjects. This allows the use of the terrain editing tools
available in Unity to generate shapes for collisions with MuJoCo. When selecting hfield type in the Unity geom
component, the right click context menu provides utility to add the corresponding Unity terrain to the scene. The data
from the terrain is dynamically kept in sync with the simulation.

MuJoCo plugins
______________

The current version of the Unity package does not support loading MJCF scenes that use :ref:`MuJoCo plugins<exPlugin>` such as
`elasticity <https://github.com/google-deepmind/mujoco/tree/main/plugin/elasticity#readme>`__ . Adding basic functionality to do this will be part of an upcoming release.

Interaction with External Processes
___________________________________

Roboti’s `MuJoCo plug-in for Unity <https://roboti.us/download.html>`_ steps the simulation in an external Python
Roboti’s `MuJoCo plug-in for Unity <https://roboti.us/download.html>`__ steps the simulation in an external Python
process, and uses Unity only for rendering. In contrast, our plug-in relies on Unity to step the simulation. It should
be possible to use our plug-in while an external process "drives" the simulation, for example by setting ``qpos``,
calling ``mj_kinematics``, synchronizing the transforms, and then using Unity to render or compute game logic. In order
to establish communication with an external process, you can use Unity's `ML-Agents
<https://github.com/Unity-Technologies/ml-agents>`_ package.
<https://github.com/Unity-Technologies/ml-agents>`__ package.