Roadmap: Baseline Health, Honest Documentation, and Feature Completion

[CodersAcademy006]

Roadmap: Baseline Health, Honest Documentation, and Feature Completion

Status: Open
Owner: maintainers
Labels: enhancement

This roadmap is the single canonical reference for ThinkRL's current state and what needs to happen next. It is split into three tracks: (1) inventory of what already exists, (2) current blockers that prevent a trustworthy baseline, and (3) the work items required to reach a clean first release.

---

1. Inventory: What Already Exists

Verified against the codebase on main (commit 3121e04).

Algorithms

Algorithm	Status
PPO (ppo.py)	Implemented
GRPO (grpo.py)	Implemented
DAPO (dapo.py)	Implemented
VAPO (vapo.py)	Implemented
COPO (copo.py)	Implemented
REINFORCE (reinforce.py)	Implemented
REINFORCE++ (reinforce_pp.py)	Implemented
IPO (ipo.py)	Implemented
PRIME (prime.py)	Implemented
STaR (star.py)	Implemented
DR-GRPO (dr_grpo.py)	Implemented
PAPO (papo.py)	Present, multimodal training incomplete
ORPO (orpo.py)	Stub: raises NotImplementedError
KTO (kto.py)	Stub: raises NotImplementedError
RLOO (rloo.py)	Stub: raises NotImplementedError

Trainers

• SFTTrainer with real train() loop.
• GRPOTrainer with real train() loop.
• ReinforcePPTrainer with real train() loop.
• STaRTrainer, CoTTrainer, ToTTrainer, MultimodalTrainer exist; completeness varies.
• Support utilities: KLController, MixedPrecisionTrainer, RewardProcessor, GAE/group-advantage helpers in training/rl_utils.py.

Models and Losses

• Actor, Critic, RewardModel, PRM (Process Reward Model).
• models/loss.py is 1,467 lines with 15+ loss implementations.
• PEFT/LoRA integration with for_llama / for_qwen presets and merging.

Infrastructure

• DeepSpeed ZeRO-2 / ZeRO-3 strategy wrappers.
• vLLM integration (integration/vllm_client.py, vllm_worker.py).
• Sequence packing, GPU-accelerated metrics (CuPy), checkpointing (SafeTensors), logging (W&B, TensorBoard, Console).
• Configuration system with YAML support and per-algorithm config dataclasses.
• Reward scoring: rewards/universal.py UniversalReward (math / code / text with <think> / <answer> structure validation).
• Sphinx documentation at docs/.
• CI workflows, pre-commit hooks, Ruff / mypy / Bandit configuration.

---

2. Current Blockers

These prevent a trustworthy, reproducible baseline. They should be resolved first.

B1. Test suite cannot collect out of the box

pyproject.toml sets asyncio_mode = "auto", but pytest-asyncio is not installed and is missing from requirements-ci.txt. Fresh environments fail collection with an INTERNALERROR before any test runs.

Fix: add pytest-asyncio>=0.23.0 to requirements-ci.txt and install it.

B2. Environment drift breaks tokenizer tests

The working venv runs transformers 5.14.1, but requirements pin transformers>=4.35.0,<5.0.0. The newer tokenizers emits a DeprecationWarning that pytest's filterwarnings = ["error"] promotes to a fatal error. Result: 4 failed + 16 errors, all in tests/test_utils/test_tokenizer.py.

Fix: pin the dependency set consistently (see F6), and scope the warning filter to tokenizers deprecations.

B3. CHANGELOG overclaims implemented features

CHANGELOG.md lists ORPO, KTO, and RLOO as added/implemented. They are stubs that raise NotImplementedError. Release notes must match reality before a public release.

Fix: mark these as planned in CHANGELOG, or implement them.

B4. CLI advertises training it cannot perform

thinkrl train|generate|sft|dpo|ppo|reward|orpo|kto parse arguments then hit # TODO: Implement actual training loop. Only grpo, star, and reinforce_pp are wired to real trainers. Users following the README get silent no-ops.

Fix: either wire the CLI commands to the existing trainers or remove/disable them until ready (see F1).

B5. Stray files in repository root

my_reward.py and reward_config.py are dev artifacts committed to the repo root. They are picked up as package files during installation.

Fix: move to examples/ or delete.

B6. ARCHITECTURE.md is stale

It states "No trainers yet" and recommends removing reasoning/, evaluation/, and registry/, but these directories now contain implementations. The architecture doc must describe the current tree.

Fix: rewrite the audit section to match the actual codebase.

---

3. Work Items

Work items are grouped by priority. Suggested labels are listed after each item.

Phase 1: Trustworthy baseline (P1)

• F1. Wire or remove CLI stubs. Connect sft, ppo, dpo, and reward to existing trainers, or gate them behind --help until implemented. labels: enhancement, help wanted
• F2. Consolidate CI workflows. Twelve workflows overlap (tests.yml, tests_core.yml, tests_latest.yml). Collapse to one test matrix plus dedicated lint/security/docs jobs. labels: enhancement
• F3. Truth audit of README and CHANGELOG. Align "Implementation Status" tables with the inventory above; redirect upstream ellanorai links and badges if this repo becomes the source of truth. labels: documentation
• F4. Enforce coverage floor. Coverage config has fail_under = 0 while the Makefile references 58%. Set a real floor and fix the gap. labels: enhancement, good first issue
• F5. Reproduce CI locally. Document a one-command local setup (Makefile targets exist but are not aligned with requirements-ci.txt). labels: documentation, good first issue

Phase 2: Feature completion (P2)

• F6. Lock the environment. Add a lockfile (uv.lock or pip-tools) so a fresh clone reproduces the passing state deterministically. labels: enhancement
• F7. Implement ORPO, KTO, RLOO. The algorithms have config scaffolding and tests in places; complete the core implementations to match CHANGELOG claims. labels: enhancement
• F8. Complete GRPO/SFT/PPO CLI end-to-end examples. Ship runnable training scripts in examples/ with configs, matching the README Quick Start. labels: enhancement, help wanted
• F9. Finish PAPO multimodal training. Vision-language path is scaffolded but not complete. labels: enhancement
• F10. Evaluation harness. README lists this as planned. Integrate lm-eval-harness or a native evaluator so trained checkpoints can be scored. labels: enhancement

Phase 3: Scale and release (P3)

• F11. Model serving. Add a serving/export path (e.g., vLLM serving integration) so trained adapters are deployable. labels: enhancement
• F12. PyPI release. README states the package is not on PyPI. Prepare packaging metadata, license, and a 0.1.0 release after Phase 1 blockers clear. labels: enhancement
• F13. CoT / ToT trainer hardening. Complete and test the reasoning trainers; document their use with reasoning configs in configs/reasoning/. labels: enhancement
• F14. Distributed training e2e. Validate the DeepSpeed path with a multi-GPU smoke test, not just unit tests. labels: enhancement

---

4. Definition of Done for This Roadmap

• Test suite collects and passes cleanly on a fresh environment (B1, B2 resolved).
• CHANGELOG, README, and ARCHITECTURE.md match the codebase (B3, B6, F3 resolved).
• Every CLI command either trains something real or is clearly marked unavailable (B4, F1 resolved).
• No stray files in the repo root (B5 resolved).
• Coverage floor enforced (F4 resolved).

Progress on this roadmap should be tracked by linking child issues back to this one.