CS 8803 DRL Final Project: Multi-Agent SoccerTwos

This repository contains the training scripts, environment wrappers, and evaluation code for the CS 8803 Deep Reinforcement Learning final project. The submitted agent utilizes Multi-Agent Proximal Policy Optimization (MAPPO) with self-play and dense reward shaping to achieve highly coordinated spatial play, defeating the official TA baseline with an 8-1-1 record.

---

For the Grader / TA: Where to Find Our Custom Code

To facilitate grading, here is a quick guide to our primary contributions:

• Reward Shaping & Observations: Look in soccer_env_wrapper.py. The top of the file contains our custom shaping coefficients, and the _shape() function contains the logic for our contact spike, anti-swarm, and marking mechanics.
• Self-Play & Training Loop: Look in train_final.py. We customized the SelfPlayUpdateCallback to track true goals (ignoring shaped rewards for metrics) and implemented a rotating opponent bank via policy_mapping_fn.
• Evaluation Script: Look in scripts/eval_checkpoint_vs_baseline.py. We implemented dynamic observation routing to automatically detect if the baseline policy requires 336 or 356 dimensions, allowing seamless head-to-head evaluation.

---

Agent Architecture & Design

The submitted agent is trained using Proximal Policy Optimization (PPO) via the Ray RLlib framework.

• Algorithm & Self-Play: PPO was utilized with a custom self-play mechanism. The agent trained against a rotating historical opponent bank (opponent_1, opponent_2, opponent_3) to prevent catastrophic forgetting.
• Neural Network: Fully connected network with hidden layers [512, 512, 256], ReLU activations, and shared value-function layers (vf_share_layers: True).
• Observation Augmentation (356-dim): The base 336-dim observation space is augmented with 20 additional dimensions (EXTRA_DIMS = 20). This explicitly tracks the position and velocity vectors of the agent, the ball, the teammate, and both opponents.
• Reward Shaping: The agent relies on a tightly constrained spatial reward system:
  • Offensive Drive: A strict +0.001 contact spike (dist_to_ball < 0.75) for maintaining physical possession to break the passive "hovering" exploit.
  • Defense & Spacing: An ANTI_SWARM_COEF penalizes teammates closer than 2.5 units to force labor division, combined with a MARKING_COEF that dynamically pulls defenders toward the closest opponent.

---

Core Files & Usage

1. The Environment Wrapper

File: soccer_env_wrapper.py This file contains the RewardWrapper and ObsWrapper classes. It is automatically applied to the Unity environment via utils.py during both training and evaluation.

2. The Training Script

File: train_final.py This script orchestrates the Ray RLlib PPO training process. It logs custom metrics (wins, losses, goals, true episode length) to a run-specific CSV file (training_log_<RUN_NAME>.csv) to track true tactical convergence.

• Usage: python train_final.py

3. The Submitted Agent (Checkpoint)

Directory: my_agent/ The submitted agent loads the exported PyTorch weights from my_agent/checkpoint.pth. This file is used by my_agent/agent.py when instantiating FinalAgent.

---

How to Run the Evaluation

To reproduce the evaluation results against the CEIA or TA baseline, use our custom dynamic evaluation script. This script runs headlessly at 20x time-scale and outputs a final Win/Loss/Draw summary.

1. Ensure your conda environment is active.
2. Run the dynamic evaluation script (replace the checkpoint paths if necessary):

python scripts/eval_checkpoint_vs_baseline.py --checkpoint trained_model/checkpoint_YOUR_BEST_CHECKPOINT --baseline path/to/ta_baseline_checkpoint --episodes 10 --port 6100

(Note: The evaluation script automatically routes either the base 336-dim array or the augmented 356-dim array to the opponent depending on its expected tensor shape).