Dressage MOPD trains one Megatron student from multiple frozen Megatron teachers on the same actor GPUs. Every teacher and the student must have the same architecture, tokenizer, vocabulary, and token IDs.
Teachers are not separate services. During actor initialization, each teacher
checkpoint is loaded once and copied into Slime's existing pinned-CPU
TensorBackuper. For every local training batch the actor groups samples by
teacher_id, restores one teacher onto the shared GPU model buffers, scores
that teacher's subset, and then restores the student before training. GPU model
memory is reused; CPU memory grows with the number of teachers.
For each response token sampled on-policy from the student, Dressage computes the stopped-gradient advantage
A_t = clip(log pi_teacher(y_t) - log pi_student_old(y_t), -5, 5)
and minimizes the direct policy-gradient loss
L = -mean(A_t * log pi_student_current(y_t)).
This is the paper-standard MOPD objective. Environment reward remains available for monitoring, but is not read by the advantage or loss. There is no GRPO return, PPO ratio/clipping, value target, entropy bonus, or additive AT-KL/OPD term. The public launcher fails closed on N greater than one and on options that are incompatible with the dedicated MOPD objective.
The figure below is a historical 20-step run on ALFWorld and HotpotQA with seed 1234. It is retained as a routing and systems snapshot; its objective metrics are not directly comparable with the dedicated MOPD path documented below. Dark lines are centered five-step rolling means, and light lines are raw step values.
The two domains show different distillation dynamics. HotpotQA's routed MOPD K1 falls from about 0.22 to 0.04, while ALFWorld starts much lower and stays between 0.003 and 0.005. Global routed K1 declines steadily, and both trainable trajectory reward curves finish above their early values. Gradient norm decreases without a sustained spike, while policy entropy and the train-rollout log-probability gap remain stable.
This run is primarily a mechanism and stability check: it confirms that both teachers remain active and that distillation proceeds without destabilizing training. Because it covers one seed with changing training prompts and has no matched control, it should not be read as a held-out capability comparison.
The implementation uses:
Sample.generate_function_pathfor per-dataset rollout dispatch;- Dressage's existing custom train-data converter to place
teacher_idin Slime's native train-sidepromptpassthrough field; - upstream
create_training_models(..., actor_cls=...)to install the Dressage-owned rotating actor without monkey-patching a Slime module; - Slime's
TensorBackuper, checkpoint loader,compute_log_prob, DP partitioning, and custom advantage/loss hooks.
The small dressage.training.mopd_train driver mirrors upstream train.py
because upstream exposes actor_cls at the model factory but not yet on the
stock CLI. Compatibility tests detect drift in this factory contract.
There is no MOPD code under dressage/rollout/generate and no Slime source
patch.
Start from
examples/data/mopd/mopd_alfworld_hotpotqa.example.json. Important fields:
teachers.<id>.load: frozen Megatron checkpoint root;teachers.<id>.ckpt_step: optional checkpoint iteration;datasets[].teacher_id: authoritative teacher for that dataset;datasets[].weight: smooth weighted-round-robin sampling weight;datasets[].agent_mode:blackboxorwhitebox;datasets[].generate_function_path: required for whitebox data and optional for a specialized blackbox implementation;reward_modules: task reward registration modules;runtime_env_keys: task-specific environment variables copied to Ray.
There is no domain router or default teacher. The data source writes direct
metadata["teacher_id"] and the native Sample.generate_function_path.
Conversion validates the route and fails before training if multi-segment
siblings disagree.
For every DP-local batch:
- Decode one teacher ID per sample from the native train-data passthrough.
- Build compact dynamic microbatches for the first distinct teacher.
- Restore that teacher from pinned CPU memory to the shared model buffers.
- Compute response-token log-probabilities only for its routed samples.
- Repeat for other distinct teachers and scatter results to original order.
- Restore the student/old actor and compute its response-token log-probability.
- Compute the stopped teacher-minus-old-student advantage and apply the direct
-A_t * log pi_student_currentloss.
The launcher intentionally uses Dressage's custom objective hooks:
--loss-type custom_loss
--custom-advantage-function-path dressage.training.mopd_loss.compute_mopd_advantages
--custom-loss-function-path dressage.training.mopd_loss.mopd_policy_loss
MOPDMegatronTrainRayActor loads all configured named teachers independently
of Slime's generic --use-opd switch. The MOPD entrypoint labels the running
estimator mopd after upstream argument parsing; no Slime source patch is
required.
export DRESSAGE_MOPD_TEACHER_CONFIG=/path/to/mopd.json
TP_SIZE=4 \
CP_SIZE=1 \
ROLLOUT_BATCH_SIZE=16 \
N_SAMPLES_PER_PROMPT=1 \
GLOBAL_BATCH_SIZE=16 \
bash examples/scripts/run_mopd_qwen3.5_sync.shEnable W&B without placing credentials in the trainer command line:
USE_WANDB=1 \
WANDB_PROJECT=slime-dev \
WANDB_GROUP=mopd-alfworld-hotpotqa \
bash examples/scripts/run_mopd_qwen3.5_sync.shIn addition to Slime's global training metrics, Dressage logs the direct loss,
sampled-token reverse-KL estimate, advantage statistics, and within-update KL
under train/mopd_*. It also logs per-teacher trainable-trajectory reward and
sampled-token reverse-KL curves under
rollout/mopd/raw_reward_trainable_trajectory_mean/<teacher_id> and
rollout/mopd/opd_reverse_kl_train_aggregation_mean/<teacher_id>.
SEED and ROLLOUT_SEED are forwarded explicitly by the launcher.
No teacher process is started separately. Checkpoint paths are validated by the launcher, and all teacher loading happens inside the student actor group.