Ideas braindump

[rosikand]

Distillation:

allocate part of training time to train a bigger > 16mb model --> distill into student.

• The core idea: use part of your 10-minute budget to train a larger model (or a wider/deeper version of your model that exceeds 16MB), then spend the remaining time distilling it into the 16MB-compliant student. You're effectively laundering extra capacity through the training process. The soft targets from the teacher carry richer information than the hard token labels — dark knowledge about which wrong tokens are "almost right" — and this is well-established to help small models punch above their weight. The practical version that fits the rules would be something like: spend minutes 0–6 training a 32MB model on FineWeb, then minutes 6–10 training the 16MB student on a mix of hard cross-entropy and KL divergence against the teacher's logits. Everything runs in a single train_gpt.py, fully reproducible. No external data, no pre-computation — just two phases in one script.
• I also wonder if you can do test-time-distillation (TTD): you have 10 minutes of eval time so obviouslly some form of TTT will win and take advantage of that. Can we do this larger-to-smaller distillation at test time as well?

Systems-side efficiency

• Systems optimizations basically can get you as close to the lower bound as possible for your given algorithmic design.
• Helpful to first find good algorithm --> then optimize the system.
• might be a good idea to run > 10 min or > 16mb for testing out algorithm ideas --> find a good one --> systems optimize it down to the limits.
• Another perspective: doing intense systems optimizations will allow you to fit the maximum size model, as fast as possible.
• Two sides to optimize: model size, model throughput: (1) self-explanatory; bigger model, more capacity to learn representations, (2) means we need to maximize the inference/forward pass of the model to pump out as many gradient steps as possible.

Quantization

• Continuing from above, this is the highest rate of return for a systems side optimization.
• Quantize large model --> fit it into the 16mb.
• int6 or int4 quantization → pack more parameters → combine with all the eval tricks (sliding window, long context)

The 1-Bit Paradigm: BitNet b1.58 and Ternary Networks

Fused Triton Megakernels

• we only have 600 seconds for training. pytorch overhead will be too much.
• interface directly with the silicon execution units of the 8xH100 SXM cluster.

Eval tricks

Sliding window

• Baseline chops validation into non-overlapping 1024-token chunks, so tokens get 0–1023 context (avg ~512)
• Sliding window with stride=64 scores only the rightmost 64 tokens per window, giving every token 960+ context
• This alone is worth ~0.032 BPB improvement with zero training changes
• Eval time goes from ~16s to ~70s on 8×H100 — well within the 10-min eval budget
• The LoRA TTT ablation showed that most of TTT's apparent gains were actually just the strided eval, not the gradient steps
• Key params: EVAL_STRIDE=64, EVAL_BATCH_SEQS=1024 (batch windows for GPU utilization)
• Smaller stride = more context per token but slower eval; stride=64 seems like a good tradeoff given the time budget
• This is now table stakes — every competitive submission will include it

Long context

• Training at seq_len 2048 or 4096 instead of 1024 helps the model learn longer-range dependencies
• 4k seq length + tuned hyperparams got 1.2014 BPB (before sliding window eval was even applied on top)
• Tradeoff: longer sequences = fewer steps in the 10-min budget (each step is slower), so you need to compensate with batch size / LR adjustments
• The warmdown submission found that well-trained models (many steps) benefit less from aggressive length extrapolation at eval — eval@1408 (1.375×) was optimal vs eval@2048 hurting
• NTK-RoPE can extend effective context at eval time beyond training length, but gains depend on how well-trained the model is
• Combining train@2048+ with sliding window eval at that longer length is likely a strong combo that nobody has fully exploited yet
• Batch token budget may need reduction (the 4k submission used 393k tokens/step instead of 524k) to fit longer sequences in memory

Born-Again Networks

• train the 16MB model, then re-initialize and train a fresh 16MB model using the first one as teacher. Multiple generations of this can sometimes improve results. The question is whether you have time budget for multiple generations.

Ensembles

• This is obvious from the kaggle days.
• Of course, compute is a problem

Data augmentation

• Obvious one. Expand docs via data aug. see recent megadocs paper.
• big question whether the augmentation can even be useful in the 10 minutes we have for training.
• test time augmentation?

prefix cost

• Recent approach on leaderboard stores weights in the prefix. so the cost is eaten there. but could be better?

[rosikand]

• You also have 8 gpu's on train and eval time... need to take advantage of that. Why would you need 8 gpu's for eval and 10 min? They almost want you to do something with that excess time and compute.

[rosikand]

Test-Time Distillation (TTD)

store a compressed model, algorithmically expand it at eval time, use the expanded version as a teacher to improve scoring.

---

Test-Time Distillation (TTD)

Setting

You have a fixed artifact of size $S$ bytes (e.g., 16MB) containing model weights $\theta_S$ and code. At evaluation time you have access to $C$ compute (e.g., 10 minutes on 8×H100s) and an unlabeled evaluation set $\mathcal{X}{eval}$. You are scored on $\text{BPB}(\mathcal{X}{eval})$. No training data may be accessed during evaluation.

Observation

The artifact size constraint bounds the model at rest, not the model in memory at inference. Evaluation compute $C$ is much larger than what naive inference requires. These two facts create an arbitrage: you can store a compressed representation of a model that, when expanded at eval time, yields a larger and more capable model than $S$ bytes would normally allow.

Method

TTD proceeds in three phases during evaluation:

Phase 1 — Expansion. Deterministically construct a larger "teacher" model $\theta_T$ from the stored weights $\theta_S$ using parameter-free or code-defined operations. Possible expansion operators include:

• Layer untying: If $\theta_S$ uses weight-shared (recurrent) layers, unroll them into distinct layers and perturb with a stored or deterministic noise schedule
• Rank inflation: If $\theta_S$ stores low-rank factors $A \in \mathbb{R}^{d \times r}, B \in \mathbb{R}^{r \times d}$, compute $W = AB$ and optionally add structured perturbations
• Width expansion: Duplicate neurons/heads with sign-flip symmetry breaking (à la Net2Net), producing a wider function-preserving initialization
• Depth multiplication: Stack copies of layers with residual scaling, increasing effective depth

The expansion is deterministic and defined entirely in code (which counts toward the $S$ byte budget, but code is tiny relative to weights).

Phase 2 — Transductive adaptation. Run the expanded teacher $\theta_T$ over $\mathcal{X}_{eval}$ using a self-supervised objective (next-token prediction on the evaluation set itself). This is legal because you are not accessing training data — you are adapting to the structure of the evaluation distribution. This phase can use:

• Full SGD/Adam on all expanded parameters
• LoRA adapters on the expanded model for memory efficiency
• Multiple passes over $\mathcal{X}_{eval}$ if compute allows

After adaptation you have $\theta_T'$, a teacher tuned to the eval distribution.

Phase 3 — Scoring (with optional distillation). Two sub-strategies:

• Direct scoring: Simply score $\mathcal{X}_{eval}$ with $\theta_T'$. This is simplest and works if $\theta_T'$ is fast enough to do inference within the remaining compute budget.
• Student distillation: If $\theta_T'$ is too expensive for full sliding-window inference, distill it into a smaller student $\theta_S'$ (initialized from $\theta_S$) using $\theta_T'$'s soft targets on $\mathcal{X}_{eval}$, then score with the fast student. This trades some quality for inference speed, allowing tighter sliding window strides.

Why this might work

The core claim is that $\text{BPB}(\theta_T') < \text{BPB}(\theta_S)$ because:

1. More parameters → lower loss floor. Scaling laws tell us $L(N)$ decreases with parameter count $N$. The expanded model has more effective capacity.
2. Transductive adaptation exploits eval-set structure. Even without labels, next-token prediction on the eval set adapts the model to its specific domain distribution, reducing distributional mismatch.
3. Expansion from a trained seed ≠ random initialization. The expanded model starts in a good basin because it's function-preserving (or nearly so) with respect to the trained compact model.

Why it might not work

• Expansion may not preserve quality. Naive duplication/perturbation could land in a worse basin than the original compact model, and there may not be enough eval-time compute to recover.
• The eval set may lack sufficient signal. If the 50k validation documents are too diverse for transductive adaptation to help (each domain is a tiny fraction), the adaptation could be noise.
• Compute budget tension. Expansion + adaptation + scoring must all fit in 10 minutes. Each phase competes for the same budget. The optimal allocation is unclear.
• The LoRA TTT result is discouraging. Per-document TTT only gave 0.003 BPB beyond sliding window eval. TTD would need the expansion + distribution-level adaptation to yield substantially more than per-document adaptation did.

Relation to prior work

• Test-time training (Sun et al., 2020): Adapts a fixed model with self-supervised objectives at test time. TTD extends this by also expanding the model before adaptation.
• Net2Net (Chen et al., 2015): Function-preserving network expansion for faster training. TTD repurposes expansion as a test-time capacity trick.
• Transductive learning (Vapnik, 1998): Using test inputs to improve predictions. TTD combines transduction with model expansion and optionally distillation.
• Universal Transformers (Dehghani et al., 2018): Variable-depth recurrence at test time. TTD generalizes beyond depth to width, rank, and layer diversity.

No prior work combines artifact-constrained storage → deterministic expansion → transductive adaptation → distilled scoring in a single pipeline, to our knowledge.

Open questions

1. What is the optimal expansion operator? Layer untying from a recurrent architecture seems cheapest since you need recurrence during training anyway for parameter efficiency.
2. What is the optimal compute allocation across the three phases?
3. Does distribution-level adaptation (all documents jointly) outperform per-document adaptation (which the LoRA TTT result suggests has low marginal value)?
4. Is the direct scoring path always better than student distillation, or does distillation win when the expanded model is too large for dense sliding-window inference?

[rosikand]

non-transformer approaches

e.g., use SSM models, linear attention.

[rosikand]

two model ensemble

The artifact is code + model. But the rules say nothing about having one model. What if you store two different 7MB models — one wide+shallow, one narrow+deep — and ensemble their log-probs at eval time?
log_p_final = 0.5 * log_p_model_A + 0.5 * log_p_model_B
Ensembles are one of the most reliable techniques in ML. Two models with different inductive biases make different errors. Their errors are partially uncorrelated, so the ensemble BPB is lower than either alone.
You'd train them jointly in one script (both fit in GPU memory), serialize them both into the artifact, and at eval time load both and average predictions. The question is how much BPB you lose from each model being half-sized. If the ensemble gain outweighs the individual quality loss, you win.

[rosikand]

spec decoding

can take a look into faster spec decoding algorithms like ssd

[rosikand]

Mixture of Depths (MoD) instead of MoE

Skip layers dynamically per-token using a learned routing scalar. Tokens that are "easy" (common n-grams) exit early, saving compute for hard tokens. This is
different from the Universal Transformer recurrence in PR openai#216 — it's the opposite: fewer layers for easy tokens. The saved FLOPs during training let you train
a deeper model (say 14L instead of 11L) within the same 10-min budget.

[rosikand]

Progressive quantization during training

Train in FP16 for 70% of steps, then INT8 for 20%, then INT6/INT5 for the final 10%. Each transition uses a short cosine warmup.

[rosikand]

Data-aware layer freezing

After 60% of training, freeze the bottom 4 layers and double the LR on the top 4. The bottom layers learn syntax/structure early and stabilize; the top layers
need more capacity for semantics. This effectively gives you 1.4x the training budget on the layers that matter most for BPB, at zero extra cost.

Synthetic pre-distillation

Use the first 2 minutes of training to train a tiny 3L model, then use its logits as soft targets (distillation) for the main 11L model for the remaining 8
minutes. The small model is cheap and provides a curriculum signal that speeds up early convergence of the big model. The small model is discarded — no artifact
cost.

[rosikand]

out of the box ideas

1. Train a Compiler, Not a Language Model

Don't train a transformer to predict tokens. Train it to output a program that predicts tokens — a tiny bytecode VM whose "weights" are the program
instructions. The transformer is the compiler; it runs once during training to emit a compact prediction program. At eval time you run the program, not the
transformer.

Why it's interesting: the 16MB limit constrains weights, but a program can encode arbitrary computation patterns that would take billions of parameters to
represent as a weight matrix. A 500KB program could implement specialized prediction rules for thousands of common patterns.

2. Metamorphic Weights

Train the model so its weights are the compressed representation of themselves. The idea: the model's forward pass, when run on a special input sequence,
reconstructs its own weights at higher precision. You store INT4 weights in the artifact, but at eval time the model "decompresses itself" by running inference
on its own weight matrices to recover INT8+ effective precision.

Why it's interesting: it turns the model into its own decompressor. The 16MB limit applies to stored weights, not to the effective precision at runtime. This is
essentially learned neural compression applied reflexively.

3. Evolution During TTT

Instead of gradient-based LoRA TTT, maintain a population of 4-8 LoRA weight vectors and do evolutionary selection per chunk. Score each variant on the current
chunk (legal — you're scoring before selecting), keep the best 2, mutate to refill the population. No gradients, no learning rate, no optimizer state.

Why it's interesting: evolutionary TTT has zero hyperparameter sensitivity (no LR, no momentum, no weight decay), adapts at a fundamentally different timescale
than SGD, and nobody in ML does online evolution anymore. It's also trivially parallelizable across the 8 H100s during eval.

4. Prediction Market Ensemble

Run 3 different tiny models in parallel during eval (they fit in 16MB together at INT4). Each model places a "bet" (logit distribution) on each token. A
market-making algorithm (not learned — pure math, like a logarithmic market scoring rule) aggregates bets, and models that predict well accumulate influence.
Bad models get automatically downweighted.

Why it's interesting: it's an ensemble that requires zero training to combine — the market mechanism is the aggregation. It naturally handles distribution shift
(a model good at code gets upweighted in code sections). And you can pitch it as "decentralized intelligence" in your README.

5. Linguistic Priors as Architecture

Hardcode syntactic structure into the attention mask. English has known dependency patterns — determiners attend to nouns, verbs to subjects, etc. Use a tiny
POS tagger (or just regex heuristics) to build a linguistically-informed sparse attention pattern as a bias term. The model can override it, but it starts with
human knowledge about language structure baked in.

Why it's interesting: everyone treats the transformer as a blank slate. But you have 10 minutes of training — giving the model a head start with linguistic
inductive biases could save thousands of gradient steps. It's the kind of idea that's "obvious in retrospect" and shows you think about what transformers are
actually learning.

[rosikand]

• diffusion models
• self-distillation
• online distillation (co-training student / teacher)

I think I will also do an experiment for the non-record on distillation, trying out a bunch of different techniques, talking about the methods, and writing up the results.