feat(examples): compare trained routers by answer quality and cost

[svonava]

Adds examples/train-a-router, a standalone walkthrough for lowering answer-model spending while measuring the correct answers lost versus always using the stronger model. Learners establish baselines, adapt Laya with LoRA, train small classifiers on fixed Qwen embeddings, compare teacher-generated training labels, and route new requests through SIE Cloud or their own server. The original request-routing example stays separate.

The main study uses 2,000 training questions, 1,000 calibration questions, and 3,000 separate test questions, grouped by detected question families. Training chooses model settings; calibration chooses routing cutoffs; saved decisions are evaluated on the test answers. At the primary one-point calibration setting:

Router	Savings	Test quality gap
Unchanged Laya	21.6% before hosting	1.50 points
Laya + LoRA	16.4% before hosting	0.80 points
Qwen embeddings + classifiers	20.9%, including embeddings	0.47 points
Teacher-label classifiers, 1,539 examples	19.7%, including embeddings	0.33 points

LoRA first preserves more answer quality; embeddings then improve both measured cost and quality. Teacher labels are an alternative source of training supervision, checked against a same-question reference-label control. The separately declared secondary embedding rule saves 33.8% with a 0.83-point test gap, reducing the estimate from $9.51 to $6.30 per 10,000 questions. Its gap interval is 0.33–1.30 points; the primary result remains unchanged.

An optional Sol/Luna case study reuses all three frozen routers on the same 3,000 questions without retraining or changing cutoffs. The embedding rules save 29.8% with a 0.27-point gap at the original primary setting and 46.2% with a 0.43-point gap at the secondary setting. The latter costs $3.99 versus $7.41 per 10,000; its gap interval is 0.10–0.77 points. The case study includes always-Luna's roughly 93% saving / 1.25-point gap and the uncertain advantage over random routing at equal Luna shares. One interrupted Luna response affects neither primary nor secondary router result; its unknown answer and cost remain explicit in the baseline.

The README starts with the dataset, metrics, and a six-step overview, then gives commands and an explicit purpose for each stage. Every command block starts with a comment. The quickstart covers the same process; serving guidance explains GPU/CPU requirements, latency, model size, and router overhead. Large datasets and optional weights are pinned on Hugging Face.

Validation:

• Locked installation, dependency checks, Ruff, and all eight CLI help paths passed on Modal. The executable source passed 304 unit checks and five readiness checks, with one expected Apple MPS skip.
• Anonymous public fetch, replay, and preparation reproduced the saved Qwen reports. Classifier refits reproduced held-out routes and metrics.
• A fresh public CLI LoRA run on a Modal H100 selected one epoch and reproduced the saved 500-example scores and all four routing settings exactly. Verification receipts.
• The independent Sol/Luna evaluator checked physical usage, fixed route hashes, all 3,000 questions, and paired uncertainty intervals. A readable public export removes unused provider fields and reproduces every scientific value from the private originals.
• Fresh anonymous CPU replay of the pinned Sol/Luna download ran all four documented command blocks and reproduced all seven result files byte for byte in 36.7 seconds, without credentials, private inputs, GPU, or new model calls. Public verification receipt.
• The executable code passed full required CI. The final commit changes only the README, quickstart, and transfer case study; its CI run is available separately.
• Seven learner guides have 32 command blocks with descriptive comments, and their relative file links resolve.

The model components and live routed SDK request passed validation. The complete three-GPU Docker Compose deployment has not been run as one stack. Native router hosting is unpriced; all stated savings distinguish that from measured embedding charges.

Walkthrough · Quickstart · Pinned Qwen evidence · Pinned Sol/Luna evidence

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}