How v3GPT (the ~99M-param Hinglish "chad" chat companion) was evaluated, across the whole SFT lineage (v1 → v2 → v3 → v4 → v4_2 → v4_3 final), and — just as important — what the correct industry terms are for each thing that was built. Read this first; the other files are the deep dives.
rubric-scoring-and-results.md— the single-turn rubric eval in full: held-out set, decode, the v4-calibrated rubric, and the exact cross-version scorecard (v2 / v3 / v4 / v4_2 / v4_3).capability-probes.md— the NEW probes built for v4_3: multi-turn, calc-tool, and memory/recall. The crucial distinction between pattern-level coherence (data-addressable) and variable-binding / working-memory (parameter-bound).roast-battle-arena.md— the pairwise "roast battle" (chad v1-era vs GPT-5.5, judged by DeepSeek-v4-flash). Directional, not the authoritative scorecard.- DPO eval (separate doc, see
08) — the post-SFT DPO/RPO A/B is owned by folder08. Referenced here, not covered.
Model under test: chad SFT v4_3 = checkpoint runs/sft/sft_v4_final/best (1).pt, iter 575, masked val 2.8127. Decode for eval: temp 0.8, top-p 0.9, repetition-penalty 1.3, stop <|endoftext|>. Judge: DeepSeek-v4-flash (thinking:disabled, temp 0.0) with the v4-calibrated rubric. Held-out 100-prompt set (102 judged rows). Source: evals/sft_v2_100_v4_3_final/results.md + judged_v4.jsonl.
| Axis | Scale | v4_3 | Read |
|---|---|---|---|
| Relevance | 0–2 | 1.84 | beats the v2 champion's 1.82; v4 was 1.27 |
| Tone-match | 0–2 | 1.71 | the "golden rule" (match the user's energy) |
| Voice | 0–2 | 1.93 | chill Gen-Z Hinglish |
| Coherence | 0–1 | 0.99 | essentially saturated |
| Badroast | rate | 4/102 = 3.9% | roasted-without-invite; v4 was 24% |
| Reddit-voice leaks | count | 0 / 102 | structural metric |
| No clean stop | count | 0 / 102 | structural metric |
| Loopy / degenerate | count | 0 / 102 | structural metric |
| Avg reply length | words | 11.4 | (v2 ran ~38.7) |
Verdict — a decisive win. v4_3 is the first v4-line checkpoint that simultaneously (a) recovers v2-level relevance, (b) fixes the v4-line tone/badroast collapse, and (c) keeps voice + coherence near ceiling — all while being the persona the product actually wants (chill, energy-matching) rather than v2's "savage roast everything." The capability probes (multi-turn / calc / memory) are the v4_3-era stress tests and live in capability-probes.md.
Why a NEW rubric (v4 judge ≠ v2 judge). The persona changed, so the grader had to change. v2 was a savage roast bot — its judge (
judge_sft_v2.py) scoredper(persona) = "savage, funny, in-character chad roast." v4_3 is a chill, energy-matching Gen-Z companion: gentle on sad, hyped on good news, savage only when invited. Grading v4 with the v2 rubric would reward the exact cringe v4 was built to remove (it would score a greeting-roast as great persona). Sojudge_sft_v4.pyreplacedperwith tone-match + voice and added a badroast flag = "roasted/mocked a user who did not invite it" (the cardinal sin).relandcohare kept comparable across both judges.
| What was said / built | Formal term | One-line definition |
|---|---|---|
| "DeepSeek judges it" / "AI grades the bot" | LLM-as-a-judge (a.k.a. model-graded evaluation) | Using a strong LLM to score or compare model outputs in place of human raters. |
| "no right answer to check against" | Reference-free evaluation | Judging quality with no gold answer — opposite of reference-based metrics (BLEU, ROUGE, perplexity). |
| "scored it out of 2 on relevance, tone, voice…" | Rubric-graded / criteria-based scoring (MT-Bench: single-answer grading) | Score one response on explicit named axes on a fixed scale (here 0–2 per axis, coherence 0–1), no opponent needed. |
| "100 brand-new prompts it never saw" | Held-out / out-of-distribution (OOD) prompt set | Eval prompts disjoint from training data → scores reflect generalization, not memorization. |
| "roast battle, GPT-5.5 vs chad, who won" | Pairwise comparison / preference judgment, summarized as a win-rate | Head-to-head A-vs-B judging. Industry analogues: Chatbot Arena / LMSYS, MT-Bench (pairwise), AlpacaEval, Arena-Hard. |
| "judges favour whoever's listed first" | Position bias | LLM-judge failure mode: systematic preference for first-/last-listed candidate. Mitigation: randomized A/B order. |
| "judges like their own family's answers" | Self-enhancement bias | A judge over-rates its own model family's outputs. (Not a factor here — judge ≠ any contestant.) |
| "does it remember the name I told it" | In-context recall / working-memory probe | Tests whether a fact stated earlier in the same context window is retrieved later — a variable-binding capability test. |
| "does it stay on topic across turns" | Multi-turn coherence / context-carry eval | Tests whether the dialogue thread holds over several turns. |
| "Haiku filtered the Reddit lines for me" | LLM-as-judge as a data filter (a precursor, not model eval) | Same technique applied in the data-mining phase to keep only on-persona training examples. |
The judge in every v3GPT model eval is DeepSeek-v4-flash (OpenCode-Go "zen" gateway, thinking:{type:"disabled"}, temp 0.0), so all of it is automated / model-graded, with zero human raters in the scoring loop.
SFT runs were tracked with masked validation loss (a reference-based, intrinsic next-token metric). Necessary, but blind to the four behaviours that decided whether v4_3 shipped:
- Relevance — does the reply engage this message, or is it a generic line pasted under anything?
- Tone-match — does it match the user's energy (gentle on sad, hype on good news, savage only when invited)?
- Voice — is it the chill Gen-Z Hinglish friend, or has it gone bland/assistant/cringe?
- Leakage / badroast — does it break character (Reddit-comment voice) or roast someone who didn't ask?
These are behavioural, reference-free properties; a model can have great loss and still fail every one of them. That gap is exactly what LLM-as-a-judge exists to close.
| Harness | Type | Script(s) | Output | What it measures | Headline (v4_3) |
|---|---|---|---|---|---|
| Single-turn rubric | rubric-graded, reference-free, held-out | test_sft_v2_100.py (gen + auto-metrics) → judge_sft_v4.py (v4 rubric) / judge_sft_v2.py (legacy v2 rubric) |
evals/sft_v2_100_*/judged_v4.jsonl |
rel / tone / voice / coh + badroast, plus structural auto-metrics (leaks, clean-stop, loops, length) | rel 1.84, tone 1.71, voice 1.93, coh 0.99, badroast 3.9% |
| A/B between arms | position-bias-controlled pairwise | compare_v2_ab.py |
stdout / DPO docs | which checkpoint/arm is the better chad (blind, order-randomized) | base→v2 vs v3 = 70 / 32 (v3 loses) |
| Multi-turn | context-carry / multi-turn coherence | test_mt_v4.py |
evals/mt_v4_3_final/results.md |
does context carry turn-over-turn; calc fires mid-conversation | 15 convos × 3 turns: thread holds; calc fires & uses result in-convo |
| Calc-tool | tool-use mechanism + arithmetic accuracy | test_calc_v4.py |
stdout (not committed); cross-checked by diag_v4_3 |
<calc> fire rate, expression validity, in-character use of <result>, operand correctness |
mechanism 16/16; arithmetic ~9/16 (56%) |
| Memory probe | in-context recall (names) | test_memory_v4.py |
stdout (not committed) | recall a told name 1–3 turns later (still in 1024 ctx) | 0/4 genuine novel names recalled |
| Recall probe | in-context recall (fact types) + topic-continuity control | test_recall_v4.py |
stdout (not committed) | discrete-fact recall vs thread continuity | fact-recall broken / confabulates; topic-continuity control passes |
| Persona diagnostic | qualitative spread (pre-DPO) | test_diag_v4.py |
evals/diag_v4_3/results.md |
clap-back on curse/cocky, savagery bleed, calc word-problems | clap-back intact; word-problems mangle operands |
| Roast battle (v1-era) | pairwise preference arena | evals/roast_battle.ipynb |
evals/roast_battles.jsonl |
chad-v1 vs GPT-5.5, DeepSeek judge | GPT-5.5 4–0 (rounds 6 / 14) — directional |
| Phase | Judge model | How it was called | Purpose | Formal label |
|---|---|---|---|---|
| Data mining (precursor) | Claude Haiku 4.5 | Anthropic Batch API + cached rubric + structured JSON | Filter mined Reddit replies to on-persona lines (~$8 / ~109k candidates) | LLM-as-judge as a data filter |
| Model eval — single-answer (v2 rubric) | DeepSeek-v4-flash | OpenCode-Go zen gateway, thinking:disabled, temp 0.0 |
Rubric-score the held-out set on rel/per/coh | Model-graded single-answer grading |
| Model eval — single-answer (v4 rubric) | DeepSeek-v4-flash | same gateway | Rubric-score on rel/tone/voice/coh + badroast | Model-graded single-answer grading |
| Model eval — pairwise | DeepSeek-v4-flash | same gateway | Roast battle vs GPT-5.5; blind A/B between arms | Pairwise preference / arena |
The Haiku-4.5 step is curation, not model eval — same technique, applied upstream to build the training set.
"I evaluated a 99M-param Hinglish persona chatbot with LLM-as-a-judge (DeepSeek-v4-flash), because perplexity is reference-based and can't see relevance, persona, tone, or leakage. The core eval is rubric-graded single-answer scoring (MT-Bench-style) on a 100-prompt held-out set. When the persona shifted from 'savage roast bot' to 'chill energy-matching companion', I re-calibrated the rubric — replacing the old
personaaxis with tone-match + voice and adding a badroast flag for the cardinal sin (roasting someone who didn't ask). The final v4_3 model scores 1.84/2 relevance (beating the prior champion), 1.71 tone, 1.93 voice, 0.99 coherence, and 3.9% badroast (down from 24%), with zero leakage/loops/no-stops. On top of single-turn, I built capability probes — multi-turn context-carry, a calc-tool loop, and in-context memory/recall — which exposed a clean line between pattern-level coherence (the model has it) and variable-binding working-memory (a reasoning skill 99M lacks). DPO is a separate A/B on top."