Risk-Aware General-Utility Markov Decision Processes

Publication

This repository contains the official implementation of the paper "Risk-Aware General-Utility Markov Decision Processes" by Pedro P. Santos, Fábio Vital, Alberto Sardinha, and Francisco S. Melo.

Installation

1. Assuming miniconda is already installed.

2. Install python environment

conda env create -f environment.yml -y
conda activate risk_aware_gumdp
export CONDA_ENVS_PATH="<miniconda3-path>"  # normally at `<home-dir>/miniconda3/envs`
poetry config virtualenvs.path $CONDA_ENVS_PATH
poetry config virtualenvs.create false
poetry install

Run Experiments

Illustrative Environments

Standard MDP

ERM-BI - Baseline

python src/risk_aware_gumdp/simulations/erm_backward_induction.py --env=linear_mdp --H=20 --N=100 --erm_beta=0.1
python src/risk_aware_gumdp/simulations/erm_backward_induction.py --env=linear_mdp --H=20 --N=100 --erm_beta=5.0
python src/risk_aware_gumdp/simulations/erm_backward_induction.py --env=linear_mdp --H=20 --N=100 --erm_beta=10.0

ERM-MCTS - Ours

python src/risk_aware_gumdp/simulations/erm_mcts.py --env=linear_mdp --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=0.1
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=linear_mdp --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=5.0
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=linear_mdp --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=10.0

MSEE

python src/risk_aware_gumdp/simulations/erm_mcts.py --env=entropy_mdp --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=0.1
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=entropy_mdp --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=25.0
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=entropy_mdp --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=50.0

IL

python src/risk_aware_gumdp/simulations/erm_mcts.py --env=imitation_learning_mdp --H=20 --N=100 --n_iter_per_timestep=2000 --erm_beta=0.1
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=imitation_learning_mdp --H=20 --N=100 --n_iter_per_timestep=2000 --erm_beta=40.0
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=imitation_learning_mdp --H=20 --N=100 --n_iter_per_timestep=2000 --erm_beta=80.0

MO

Weighted Cost

python src/risk_aware_gumdp/simulations/erm_mcts.py --env=multi_objective_mdp_weighted --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=0.1
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=multi_objective_mdp_weighted --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=20.0
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=multi_objective_mdp_weighted --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=40.0

Max Cost

python src/risk_aware_gumdp/simulations/erm_mcts.py --env=multi_objective_mdp_max --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=0.1
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=multi_objective_mdp_max --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=10.0
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=multi_objective_mdp_max --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=20.0

Min Cost

python src/risk_aware_gumdp/simulations/erm_mcts.py --env=multi_objective_mdp_min --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=0.1
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=multi_objective_mdp_min --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=20.0
python src/risk_aware_gumdp/simulations/erm_mcts.py --env=multi_objective_mdp_min --H=20 --N=100 --n_iter_per_timestep=500 --erm_beta=50.0

Grid Environments

MSEE

./scripts/erm_mcts.sh --env=msee --h=200 --gamma=0.99 --steps=1024 --beta=0.001 --bs=128 --micro_bs=4
./scripts/erm_mcts.sh --env=msee --h=200 --gamma=0.99 --steps=1024 --beta=1.0 --bs=128 --micro_bs=4
./scripts/erm_mcts.sh --env=msee --h=200 --gamma=0.99 --steps=1024 --beta=1000.0 --bs=128 --micro_bs=4

IL

./scripts/erm_mcts.sh --env=il --h=99 --gamma=0.99 --steps=1024 --beta=0.002 --bs=128 --micro_bs=4
./scripts/erm_mcts.sh --env=il --h=99 --gamma=0.99 --steps=1024 --beta=1.0 --bs=128 --micro_bs=4
./scripts/erm_mcts.sh --env=il --h=99 --gamma=0.99 --steps=1024 --beta=500.0 --bs=128 --micro_bs=4

MO

./scripts/erm_mcts.sh --env=mo --h=40 --gamma=0.99 --steps=1024 --beta=0.125 --bs=128 --micro_bs=4
./scripts/erm_mcts.sh --env=mo --h=40 --gamma=0.99 --steps=1024 --beta=1.0 --bs=128 --micro_bs=4
./scripts/erm_mcts.sh --env=mo --h=40 --gamma=0.99 --steps=1024 --beta=8.0 --bs=128 --micro_bs=4

Generate Plots

After running the desired experiments, check the notebooks inside /notebooks to produce the respective plots.