Skip to content

Record: EGGROLL v2 — val_bpb 1.1161 (3-seed mean, std 0.0001)#1156

Open
haikosys wants to merge 1 commit intoopenai:mainfrom
haikosys:eggroll-submission
Open

Record: EGGROLL v2 — val_bpb 1.1161 (3-seed mean, std 0.0001)#1156
haikosys wants to merge 1 commit intoopenai:mainfrom
haikosys:eggroll-submission

Conversation

@haikosys
Copy link
Copy Markdown

Record: EGGROLL v2 — val_bpb 1.1161 (3-seed mean, std 0.0001)

val_bpb: 1.1161 | val_loss: 1.884 nats | ~15.3 MB | 8×H100 SXM | Legal TTT

Built on PR #1130 by @Gusanidas (Kitchen Sink V2)
Foundation: PR #549 by @abaybektursun

3-seed validation, all artifacts under 16,000,000 bytes, all training under 600s.

Results

Seed val_loss (nats) val_bpb Artifact (bytes)
42 1.8848 1.1163 15,227,040
1337 1.8844 1.1160 15,427,072
2024 1.8844 1.1161 15,398,172
Mean 1.1161
Std 0.0001

Novel Contribution: EGGROLL (Antithetic Ternary Bin Search)

Post-GPTQ quantization refinement that directly optimizes INT6 bin assignments against BPB loss during eval budget (60s).

Algorithm: For each step, pick a quantized weight tensor, select 1024 random indices, test shifting bins +1 and -1 (antithetic pair), keep whichever improves loss. Strictly additive — cannot degrade
quality.

Properties:

  • Zeroth-order optimization (no gradients) on discrete quantized indices
  • Complementary to GPTQ: Hessian-based initial quantization + evolutionary refinement
  • 6-14 bin improvements per seed
  • Runs during eval budget, zero training impact

Also adds missing eval_val_sliding_ttt call to PR #1130's eval pipeline.

Timing

Phase Duration Budget
Training (8458 steps @ 69ms) 586s 600s
GPTQ calibration 14s (within training)
EGGROLL refinement 60s 600s eval
Sliding window eval 77s 600s eval
Score-first TTT ~470s 600s eval

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

val_bpb: 1.1161 | val_loss: 1.884 nats | ~15.3 MB | 8×H100 SXM | Legal TTT

Seeds: 42=1.1163, 1337=1.1160, 2024=1.1161 | Mean=1.1161, Std=0.0001

Novel: EGGROLL Antithetic Ternary Bin Search — post-GPTQ bin refinement
Also: adds missing TTT call to PR openai#1130 eval pipeline

Built on PR openai#1130 by @Gusanidas, PR openai#549 by @abaybektursun

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@MatoTeziTanka
Copy link
Copy Markdown

Community Review — Record: EGGROLL v2 — val_bpb 1.1161 (3-seed mean, std 0.0001)

BPB: 1.1161 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA 03ea861d08ad, file records/track_10min_16mb/2026-03-30_EGGROLL_PostGPTQ_BinRefinement/train_gpt.py):

The TTT path at line 1311 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=11, vocab=1024, code=129636 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.


Reviewed by @MatoTeziTankaThe Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=11, vocab=1024, code=129636 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants