Feature/dflash training#51

Closed

zhubohao911 wants to merge 121 commits intotorchspec-project:mainfrom

zhubohao911:feature/dflash-training

Commits on Mar 19, 2026

Add DFlash training implementation

Xing Han
authored and
Xing Han
committed
Add torch dynamo config fallback for flex attention

Xing Han
authored and
Xing Han
committed
Add DFlash framework integration: config dispatch, N-layer target model, and training entry

Xing Han
authored and
Xing Han
committed
Add DFlash training quality tests, architecture validation, and GPU config

Xing Han
authored and
Xing Han
committed
Add 1-GPU training configs and RunPod training script

Xing Han
authored and
Xing Han
committed
Fix non-interactive WandB prompt and add SSH helper

Xing Han
authored and
Xing Han
committed
Use --system-site-packages venv to avoid re-downloading PyTorch

Xing Han
authored and
Xing Han
committed
Make SGLang and vLLM imports optional in engine __init__

Xing Han
authored and
Xing Han
committed
Lazy-import SGLang and vLLM engines in factory.py

Xing Han
authored and
Xing Han
committed
Remove eval_data_path from 1-GPU configs to avoid colocate timeout

Xing Han
authored and
Xing Han
committed
Fix DFlash config for Qwen3-8B and add inductor GEMM fallback

Xing Han
authored and
Xing Han
committed
Fix inductor GEMM backend and DFlash dtype mismatch

Xing Han
authored and
Xing Han
committed
Use aot_eager backend for flex_attention compilation

Xing Han
authored and
Xing Han
committed
Cast DFlash draft model to bfloat16 before FSDP wrapping

Xing Han
authored and
Xing Han
committed
Debug DFlash zero-loss bug: vectorize noise input, add anchor fallback

Xing Han
authored and
Xing Han
committed
Track dflash_implementation_log.md in git

Xing Han
authored and
Xing Han
committed
[Feature] add Minimax support and Kimi parser updates (#45)
yubofredwang
authored
Fix stale test assertion: update expected config values for Qwen3-8B

Xing Han
authored and
Xing Han
committed
Fix DFlash metric key names to match controller expectations

Xing Han
authored and
Xing Han
committed
Update implementation log: Session 6 — zero-loss bug fix and GPU results

Xing Han
authored and
Xing Han
committed
Add DFlash inference benchmark script

Xing Han
authored and
Xing Han
committed
DFlash inference: checkpoint extract, 1-GPU bench config, benchmark script

Xing Han
authored and
Xing Han
committed
Fix 5 critical DFlash bugs: 5-layer model, Q/K-norm, block_keep_mask, loss_mask, RoPE order

Xing Han
authored and
Xing Han
committed
[Docs] Add 5-node MiniMax example files (#47)

yubofredwang
and
Copilot
authored
[Docs] refresh README structure and links (#46)
yubofredwang
authored
[Docs] Add PyTorch Blog Link (#48)
yubofredwang
authored
Fix build_target_layer_ids to match SpecForge formula exactly

Xing Han
authored and
Xing Han
committed
Phase B validation: fix training hyperparams and document 4-GPU results

Xing Han
authored and
Xing Han
committed
Add min_loss_tokens filter to preprocess_conversations

Xing Han
authored and
Xing Han
committed
Add Phase C scripts: PerfectBlend data prep and full training launcher

Xing Han
authored and
Xing Han
committed

Commits on Mar 20, 2026

Fix collator crash when loss_mask length differs from input_ids

Xing Han
authored and
Xing Han
committed
Switch FlexAttention from aot_eager to inductor backend for 2-5x speedup

Xing Han
authored and
Xing Han
committed
Document aot_eager vs inductor analysis and 3x speedup results

Xing Han
authored and
Xing Han
committed
Fix minimax type mismatch (#49)
yubofredwang
authored
docker: add RunPod Dockerfile for sglang v0.5.8.post1

Xing Han
authored and
Xing Han
committed
Add checkpoint rotation to prevent disk quota exceeded during training

Xing Han
authored and
Xing Han
committed
Document Phase C crash debugging: disk quota, eval timeout, checkpoint rotation

Xing Han
authored and
Xing Han
committed
Document training speed optimization: 6.6hr → 3.1hr with batch/anchor tuning

Xing Han
authored and
Xing Han
committed
Document checkpoint backup, resume instructions, and full attempt summary

Xing Han
authored and
Xing Han
committed
Document complete pod restart recovery procedure for DFlash training resume

Xing Han
authored and
Xing Han
committed

Commits on Mar 21, 2026

Consolidate DFlash implementation log: 1697 → 588 lines

Xing Han
authored and
Xing Han
committed
Document Session 10: PyTorch 2.9.1 migration and Phase C resume from step 17001

Xing Han
authored and
Xing Han
committed
Optimize DFlash training speed: remove compile overhead and enable GQA

Xing Han
authored and
Xing Han
committed
Document Session 11: torch 2.9.1 speed regression investigation

Xing Han
authored and
Xing Han
committed
Document DFlash benchmark results (τ=1.86) and SpecForge comparison

Xing Han
authored and
Xing Han
committed
Merge branch 'torchspec-project:main' into feature/dflash-training
zhubohao911
authored
Clean up DFlash branch: remove dead eval code, add comments to shared files

Xing Han
authored and
Xing Han
committed

Commits on Mar 22, 2026

Fix DFlash training bugs and add missing test coverage

Xing Han
authored and
Xing Han
committed
dflash test plan and implementation log updates

Xing Han
authored and
Xing Han
committed
remove old implementation log

Xing Han
authored and
Xing Han
committed
Update DFlash docs: Phase 1 complete, Issue 26 resolved, Phase D results

Xing Han
authored and
Xing Han
committed
Document Phase 2 speed benchmark results: pipeline bottleneck identified

Xing Han
authored and
Xing Han
committed
Update speed benchmark with instrumented timing breakdown

Xing Han
authored and
Xing Han
committed
Add compute sub-breakdown profiling instrumentation

Xing Han
authored and
Xing Han
committed
Document compute sub-breakdown profiling results

Xing Han
authored and
Xing Han
committed
Document speed optimization test results (Section 2.5)

Xing Han
authored and
Xing Han
committed
Add three compute speed optimizations

Xing Han
authored and
Xing Han
committed
Document compute optimization results: no_sync +8%, compile not viable

Xing Han
authored and
Xing Han
committed
Document Phase F speed optimization session results

Xing Han
authored and
Xing Han
committed
Document FSDP strategy and GPU scaling benchmark results (Tests 1-3)

Xing Han
authored and
Xing Han
committed
Increase find_free_port timeout from 30s to 120s for PyTorch 2.9+ compatibility

Xing Han
authored and
Xing Han
committed
Document Test 4 results: 2-inference GPU is strictly worse (1.1 step/s)

Xing Han
authored and
Xing Han
committed
Document hardware topology analysis: NVLink unused, Mooncake TCP is bottleneck

Xing Han
authored and
Xing Han
committed
Add async data pre-fetch to overlap Mooncake transfer with GPU compute

Xing Han
authored and
Xing Han
committed
Fix PrefetchedDataFetcher: use persistent thread across training steps

Xing Han
authored and
Xing Han
committed
Fix prefetch GPU contention: stage data on CPU, move to GPU synchronously

Xing Han
authored and
Xing Han
committed
Document Test 5a/5b/6 results: CPU prefetch 6.8 step/s, NVLink attempt

Xing Han
authored and
Xing Han
committed
Document Test 6 NVLink investigation (not viable) and Test 7 (3 GPU + CPU prefetch)

Xing Han
authored and
Xing Han
committed
Update config to best training configuration and add Phase 3 τ benchmark plan

Xing Han
authored and
Xing Han
committed
Update Phase 3 plan: incremental approach, train 5k steps first

Xing Han
authored and
Xing Han
committed
Add SpecForge DFlash training reference doc

Xing Han
authored and
Xing Han
committed
Document Phase 3 status, environment issues, and known workarounds

Xing Han
authored and
Xing Han
committed
Phase 3: 2-epoch training with resume, prefetch_depth=8, auto GEMM fix

Xing Han
authored and
Xing Han
committed
Phase 3: switch to batch=2 accum=2 seq=2048 for 5.3 step/s throughput

Xing Han
authored and
Xing Han
committed
Debug: add bounds check for noise_ids in _create_noise_embed

Xing Han
authored and
Xing Han
committed
Fix: clone mooncake tensors before buffer cleanup to prevent use-after-free

Xing Han
authored and
Xing Han
committed
Fix FlexAttention recompile overflow: bucketed padding + higher recompile limit

Xing Han
authored and
Xing Han
committed
Switch to batch=1 accum=4 for better Mooncake TCP overlap

Xing Han
authored and
Xing Han
committed
Fix 5x speed regression: skip clone for batch=1 to preserve pinned memory

Xing Han
authored and
Xing Han
committed
Add RunPod setup script consolidating all env fixes from issues/commits

Xing Han
authored and
Xing Han
committed
Update Dockerfile and setup script for faster pod provisioning

Xing Han
authored and
Xing Han
committed
Add slim/full Docker image variants via INCLUDE_MODEL build arg

Xing Han
authored and
Xing Han
committed
Rewrite RunPod guide: Docker image + script-based setup

Xing Han
authored and
Xing Han
committed
Add GitHub Actions workflow for Docker image build

Xing Han
authored and
Xing Han
committed
Fix Docker build: remove [fa] extra (flash-attention-cute metadata mismatch)

Xing Han
authored and
Xing Han
committed

Commits on Mar 23, 2026

Expand benchmark to 50 diverse prompts with τ distribution analysis

Xing Han
authored and
Xing Han
committed
Document Phase 3 training results: 2-epoch run, τ=1.78 benchmark

Xing Han
authored and
Xing Han
committed
Extend training to 3 epochs, keep 2 checkpoints for comparison

Xing Han
authored and
Xing Han
committed
Add --prompt_file argument to benchmark script

Xing Han
authored and
Xing Han
committed
Update RunPod guide: Docker image, tmux, monitoring instructions

Xing Han
authored and
Xing Han
committed
Update training results: epoch 3 confirms data saturation at τ=1.85

Xing Han
authored and
Xing Han
committed
Add general-purpose DFlash training guide

authored and
Xing Han
committed

Commits on Mar 24, 2026

Commits on Mar 25, 2026

Commits on Mar 26, 2026

Merge branch 'feature/dd-test' into feature/dflash-training

Xing Han
authored and
Xing Han
committed

Commits on Mar 27, 2026

Commits on Mar 28, 2026

Commits on Mar 30, 2026

Add hyperparameter sweep infra and Phase K convergence results
xinghandd
committed

Commits on Apr 1, 2026

Commits on Apr 2, 2026