Pessimistic Agents

Pessimistic Agents are ask-for-help reinforcement learning agents that offer guarantees of:

1. Eventually outperforming the mentor
2. Eventually stopping querying the mentor
3. Never causing unprecedented events to happen, with arbitrary probability

In this repository, we investigate their behaviour in the faithful setting, and explore approximations that allow them to be used in real-world RL problems.

Overview - see individual README.md files for more detail.

---

Distributional Q Learning - dist_q_learning/

We introduce a tractable implementation of Pessimistic Agents. Approximate the Bayesian world and mentor models as a distribution over epistemic uncertainty of Q values. By using a pessimistic (low) quantily, we demonstrate the expected behaviour and safety results for a pessimistic agent.

Work	Status
Finite state Q Table proof of concept
Continuous deep Q learning implementation

---

Faithful implementation - cliffworld/

Implement and investigate a faithful representation of a Bayesian Pessimistic Agent.

Work	Status
Environment
Agent

On hold, some progress made in implementing the environment and mentor models.

---

Pessimistic RL - pessimistic_prior/

Apply pessimism approximation to neural network based, deep Q learning RL agents.

Work	Status
DQN proof of concept

---

Setup

Supported conda env

With anaconda

conda env create -f torch_env_cpu.yml