Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: update readme with continuous system details #1050

Closed
wants to merge 8 commits into from
Closed

docs: update readme with continuous system details #1050

Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4d7cad9
docs: update readme with continuous system details
RuanJohn Mar 1, 2024
3c1d614
Merge branch 'feat/remove_vampping_cont_act' into maintenance/update-…
RuanJohn Mar 1, 2024
fc5a778
docs: example on running a continuous action space system
RuanJohn Mar 1, 2024
4533408
docs: typo fix in continuous action space example
RuanJohn Mar 1, 2024
66fb2a4
Merge branch 'develop' into maintenance/update-readme-for-cont-systems
RuanJohn Mar 1, 2024
9df3bcd
docs: minor readme fix for abstraction description
RuanJohn Mar 1, 2024
7659bcc
Merge branch 'develop' into maintenance/update-readme-for-cont-systems
WiemKhlifi Mar 6, 2024
529ecc5
Merge branch 'develop' into maintenance/update-readme-for-cont-systems
WiemKhlifi Mar 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@ To join us in these efforts, please feel free to reach out, raise issues or read

Mava currently offers the following building blocks for MARL research:

- 🥑 **Implementations of MARL algorithms**: Implementations of multi-agent PPO systems that follow both the Centralised Training with Decentralised Execution (CTDE) and Decentralised Training with Decentralised Execution (DTDE) MARL paradigms.
- 🍬 **Environment Wrappers**: Example wrappers for mapping Jumanji environments to an environment that is compatible with Mava. At the moment, we support [Robotic Warehouse][jumanji_rware] and [Level-Based Foraging][jumanji_lbf] with plans to support more environments soon. We have also recently added support for the SMAX environment from [JaxMARL][jaxmarl].
- 🥑 **Implementations of MARL algorithms**: Implementations of multi-agent PPO systems that follow both the Centralised Training with Decentralised Execution (CTDE) and Decentralised Training with Decentralised Execution (DTDE) MARL paradigms with support for continuous and discrete action space environments.
- 🍬 **Environment Wrappers**: Example wrappers for mapping Jumanji environments to an environment that is compatible with Mava. At the moment, we support [Robotic Warehouse][jumanji_rware] and [Level-Based Foraging][jumanji_lbf] with plans to support more environments soon. We have also recently added support for the SMAX and MaBrax environments from [JaxMARL][jaxmarl].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while we're here can we add connector, cleaner, matrax, CVRP and gigastep? Maybe this should become a table of envs we support 🤔

Copy link
Contributor

@WiemKhlifi WiemKhlifi Mar 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

➕ Something like this with action type:

This table outlines the environments supported by our library ect.... 
| Library/Site | Environment Supported | Action Type |
|--------------|-----------------------|-------------|
| Jumanji      | LBF                   | Discrete    |
|              | RWARE                 | Discrete    |
|              | CVRP                  | Discrete    |
| Jaxmarl      | MABRAX                | Continuous  |
|              | SMAX                  | Continuous  |

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we will mention the different environments I think we need to add a comment that we are working on verifying the performance of mava on them because we may confirm that all works perfectly once we make a full benchmark 🙏

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: Since the connector env is using a CNN network we may add another column citing the corresponding network for each env and mention that for action space we need to check the corresponding ActionHead as well in the network.yaml file:

| Library/Site | Environment Supported | Action Space | Network
|--------------|-----------------------|-------------|-------------
| Jumanji      | LBF                   | Discrete    | Default (mlp)
|              | RWARE                 | Discrete    | Default
|              | Connector             | Discrete    | CNN
| Jaxmarl      | MABRAX                | Continuous  | Default
|              | SMAX                  | Continuous  | Default

- 🎓 **Educational Material**: [Quickstart notebook][quickstart] to demonstrate how Mava can be used and to highlight the added value of JAX-based MARL.
- 🧪 **Statistically robust evaluation**: Mava natively supports logging to json files which adhere to the standard suggested by [Gorsane et al. (2022)][toward_standard_eval]. This enables easy downstream experiment plotting and aggregation using the tools found in the [MARL-eval][marl_eval] library.

Expand Down Expand Up @@ -142,7 +142,7 @@ Furthermore, we illustrate the speed of Mava by showing the steps per second as

## Code Philosophy 🧘

The current code in Mava is adapted from [PureJaxRL][purejaxrl] which provides high-quality single-file implementations with research-friendly features. In turn, PureJaxRL is inspired by the code philosophy from [CleanRL][cleanrl]. Along this vein of easy-to-use and understandable RL codebases, Mava is not designed to be a modular library and is not meant to be imported. Our repository focuses on simplicity and clarity in its implementations while utilising the advantages offered by JAX such as `pmap` and `vmap`, making it an excellent resource for researchers and practitioners to build upon.
The current code in Mava is adapted from [PureJaxRL][purejaxrl] which provides high-quality single-file implementations with research-friendly features. In turn, PureJaxRL is inspired by the code philosophy from [CleanRL][cleanrl]. Along this vein of easy-to-use and understandable RL codebases, Mava is not designed to be a modular library and is not meant to be imported. Our repository focuses on simplicity and clarity in its implementations while utilising the advantages offered by JAX such as `pmap` and `vmap`, making it an excellent resource for researchers and practitioners to build upon. A notable difference between Mava and other single-file libraries is that Mava makes use of abstraction where relevant notably for network and environment creation.

## Installation 🎬

Expand All @@ -164,19 +164,25 @@ we advise users to explicitly install the correct JAX version (see the [official
To get started with training your first Mava system, simply run one of the system files. e.g.,

```bash
python mava/systems/ff_ippo.py
python mava/systems/ppo/ff_ippo.py
```

Mava makes use of Hydra for config management. In order to see our default system configs please see the `mava/configs/` directory. A benefit of Hydra is that configs can either be set in config yaml files or overwritten from the terminal on the fly. For an example of running a system on the LBF environment, the above code can simply be adapted as follows:

```bash
python mava/systems/ff_ippo.py env=lbf
python mava/systems/ppo/ff_ippo.py env=lbf
```

Different scenarios can also be run by making the following config updates from the terminal:

```bash
python mava/systems/ff_ippo.py env=rware env/scenario=tiny-4ag
python mava/systems/ppo/ff_ippo.py env=rware env/scenario=tiny-4ag
```

To toggle between continuous and discrete systems, simply select the continuous action space network head. To run the same system on an `MaBrax` environment make the follow config updates from the terminal:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to make it clear that MaBrax is continuous

Suggested change
To toggle between continuous and discrete systems, simply select the continuous action space network head. To run the same system on an `MaBrax` environment make the follow config updates from the terminal:
To toggle between continuous and discrete systems, simply select the continuous action space network head. To run the same system on a continuous environment, like `MaBrax`, make the follow config updates from the terminal:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To toggle between continuous and discrete systems, simply select the continuous action space network head. To run the same system on an `MaBrax` environment make the follow config updates from the terminal:
To toggle between continuous and discrete systems, simply select the continuous action space network head. To run the same system on a `MaBrax` environment make the following config updates from the terminal:


```bash
python mava/systems/ppo/ff_ippo.py env=mabrax network.action_head._target_="mava.networks.ContinuousActionHead"
```

Additionally, we also have a [Quickstart notebook][quickstart] that can be used to quickly create and train your first Multi-agent system.
Expand All @@ -194,10 +200,8 @@ Please read our [contributing docs](docs/CONTRIBUTING.md) for details on how to
We plan to iteratively expand Mava in the following increments:

- 🌴 Support for more environments.
- 🔁 More robust recurrent systems.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also close the issue Edan raised around this

- 🌳 Support for non JAX-based environments.
- 🦾 Support for off-policy algorithms.
- 🎛 Continuous action space environments and algorithms.

Please do follow along as we develop this next phase!

Expand Down
Loading