docs

v3GPT / "chad" — Project Documentation

What it is: a ~99M-parameter, decoder-only transformer (Llama/Mistral-style stack) built from scratch in PyTorch and trained on romanized Hinglish to be a chill, Gen-Z Hinglish chatbot persona — "chad" — that runs 100% in the browser behind a Next.js product.

Why this doc exists: a complete, numbers-first record of exactly what was built and how, so it can be recalled months later without re-reading the codebase. Every number here was verified against the actual code / logs / data / eval files, not memory.

Last full re-survey: 2026-06-24 (covers pretraining → SFT v1→v4_3 → DPO/RPO → tool-use → in-browser deployment → capability findings).

---

The whole project in one breath

Built a custom 32,000-vocab byte-level BPE tokenizer (on Apple MPS) → cleaned a ~4.13B-token Hinglish corpus (96% Reddit, from Pushshift per-subreddit torrents) → pretrained a 99M causal LM on Kaggle free TPU v5e-8 to base val ≈ 3.77 → ran six SFT iterations (v1→v4_3) to install persona, relevance, multi-turn ability, and a calc + gif tool system — the winning trick being knowledge distillation (real upvoted Reddit comments backtranslated into Q→A by GPT-5.x/DeepSeek) → did a DPO/RPO alignment pass for cleanup (leak-suppression + tone) → exported to ONNX and shipped it running in-browser via transformers.js behind a Next.js + Express/Mongo product. The hard lesson across every run: data moves voice/relevance/format/manners; nothing moves "funny", facts, math, or memory — that's the 99M parameter ceiling.

Headline numbers (authoritative)

Aspect	Value
Parameters	98,913,024 (~99M), weight-tied (123,489,024 if untied)
Architecture	RoPE · Grouped-Query Attention · SwiGLU · RMSNorm (pre-norm); d_model 768 · 12 layers · 12 q / 3 kv heads · head_dim 64 · d_ff 2048 · ctx 1024
Tokenizer	byte-level BPE, 32,000 vocab (5 special + 256 byte + 31,739 merges), 4.31 chars/token
Pretrain corpus	~4.13B tokens (packed train.bin = 4,001,020,617) · 21 GB · 46 shards
Pretrain	Kaggle TPU v5e-8, bf16, 65,536 tok/step, 61,050 steps → best val 3.7712 (ppl ≈ 43)
Best SFT	base→v4_3 (FINAL SFT): 91,313 convos / 4.14M tok, 5 epochs, best masked val 2.8127 (iter 575)
Final model	DPO/RPO on v4_3 → shipped as chad-dpo (RPO); chad-dpo450 = safer earlier snapshot
Best eval (v4_3)	relevance 1.84/2, tone-match 1.71, voice 1.93, coherence 0.99, badroast 3.9%
DPO eval (RPO)	relevance 1.86, tone 1.83, badroast 2%, off-topic/offensive leaks 13/103 → 1–2/103
Tools	calc + gif (plain BPE, no new tokens): calc mechanism 16/16, arithmetic ~9/16 (56%)
Deployment	in-browser transformers.js, ONNX fp32 472 MB (shipped — int8 119 MB wrecks calc), ~78 tok/s WASM

Heads-up on the early scaffold: the root README.md and src/v3gpt/config.py describe a 12.6M-param, 8192-vocab model. That was the initial learning scaffold — not the trained model. The real shape lives in train_tpu.py / train.py / the kernels.

The model ladder (pretrain → SFT → DPO)

Stage	Init from	Data	Best val	Eval headline	Rank
Pretrain	scratch	4.13B Hinglish tok	3.7712	fluent Hinglish, no persona	base
SFT v1	base	132,717 Reddit pairs, unmasked	3.777	wholesome, not chad	worst
SFT v2	base	21,321 generated convos, masked	2.7716	rel 1.82 / persona 1.98 — ex-champion	2nd
SFT v3	base	v2 + 492k CSV blend	2.115*	rel 1.61 — split-personality, regressed	5th
SFT v4	base	distill 35k + convo 12k + calc 6k	2.894	tools+manners win, but rel 1.27 / badroast 24%	3rd
SFT v4_2	base	distill cut to 8k	2.646*	erratic tools — backfired	4th
SFT v4_3	base	distill 35k + convo 24k + calc 15k + multi-turn 18k	2.8127	rel 1.84 / badroast 3.9% — best SFT	1st (SFT)
DPO / RPO	v4_3	14,805 generated preference pairs	—	leak-suppression + tone; shipped as chad-dpo	final

* lower val on v3/v4_2 is an easier distribution, not a quality win — judge by eval, not loss.

---

Documentation map

#	Section	Covers
01	Architecture	the from-scratch transformer — RoPE/GQA/SwiGLU/RMSNorm, exact config, parameter breakdown
02	Tokenizer	byte-level BPE, 32k vocab, 5 special tokens, 4.31 chars/token, packing
03	Data Collection	the 4.13B pretraining corpus — Reddit torrents, cleaning, Bloom dedup, rejected sources
04	SFT Data & Distillation	the backtranslation-distill + convo + calc + multi-turn pipeline; data versions v1→v4_3; the persona pivot
05	Training	the runs & loss curves — pretrain saga + every SFT run + the model ranking
06	Compute	the 3 platforms — MPS tokenizer / TPU v5e-8 pretrain+SFT / GPU-T4 DPO; the headless Kaggle CLI
07	Evaluation	LLM-as-a-judge methodology, the rubric scorecards, and the capability probes (multi-turn, calc, memory)
08	DPO / Alignment	the preference-data pipeline, the 100M DPO razor's edge, the RPO fix, the ship decision
09	Tool Use	the calculator (<calc>) multi-turn compute loop and the gif (<gif>) tool — training + runtime
10	Deployment & Inference	ONNX export, in-browser WASM runtime, the Next.js + Express/Mongo product stack
11	Limitations & Findings	the synthesis — data-addressable vs parameter-bound, the 100M ceiling, iteration lessons, scaling

Reference docs (pre-existing)

• OPENCODE_GO_DEEPSEEK_V4_HANDOFF.md — the opencode-go / codex / DeepSeek proxy mechanics.
• pulling_reddit_community_data.md — reusable Reddit-torrent runbook.
• cpt_plan.md — continued-pretraining plan notes.

---

The one finding that frames everything

Across 6 SFT versions + DPO, capabilities split cleanly into two buckets:

• Data-addressable (SFT moved these): persona/voice, relevance/input-binding, conversational coherence, manners/match-energy, output format, tool-call format, leakage suppression.
• Parameter-bound (no data ever moved these — the 99M ceiling): genuine humor, factual recall, math-from-prose, and working-memory/variable-binding (it remembers what you're talking about, never the facts you told it — fails even with the fact in-context).

DPO/RPO confirmed the corollary: it cleaned up tone and suppressed leaks but added zero capability — "data installs behavior via SFT, not via clever objectives." Full treatment in 11-limitations-and-findings.

Current status / open threads

• Shipped model = chad-dpo (RPO on v4_3), served as ONNX fp32 in-browser; chad-dpo450 is the lower-risk earlier snapshot. The 38% "haan bro / arre bhai" opener tic is the accepted residual cost of the RPO run.
• Deployment is wired but not launched — needs the user's Vercel login + AWS; remove test routes, rotate the MongoDB + Klipy keys (pasted in chat), set CORS/API URLs first.
• The multi-turn rolling-window fix is recommended but NOT shipped — the browser sends full history and 100M degrades as it grows (see 10-deployment-and-inference).
• Funny / facts / math / memory remain unsolved at 99M — the only real paths are a bigger base (1–3B) finetune, or keeping this 100M as a Hinglish voice layer and sourcing wit/facts from a bigger model or curated bank at inference.