Hxl/update mds - #17
Closed
shane-kshongmo wants to merge 33 commits into
Closed
Conversation
寻优脚本优化，在format_result的时候增加缓存
支持pp开低精度、开overlap
支持pp开低精度、开overlap
Fix/ep comm fix
2.add xlsx test
Fix DualPipeV VPP scheduling accounting
1.fix dsv4 PARAM_SCOPE miss 2.add xlsx test
寻优允许pp=1情况，增加hidden限制
Fix HC and RMSNorm sharding
bugfix： routed dtype latency in excel
优化寻优脚本，提前筛选显存超的配置，以免这些配置去构图，建模，拖慢时间
Unify Adam optimizer cost modeling across Stack A (spec-based) and Stack B (graph-based) by introducing shared primitives in training/models/. Eliminates 4 correctness gaps where Stack B underestimated DRAM traffic by 2.3×. Shared helpers: - adam_state_bytes(P): 12 B/P storage (master+m+v) - adam_step_traffic_bytes(P): 28 B/P DRAM traffic per update - adam_step_time_s(P, hbm_bw, gpu_name, efficiency_override): memory-bound step time with size-bucketed efficiency curve - adam_params_on_rank(...): TP→PP→EP→ZeRO sharding with dual-view semantics (apply_dp_for_zero knob for storage vs step views) Stack A refactoring: - _compute_optimizer_time now thin wrapper calling shared helpers - Eliminates duplicate _ADAM_UPDATE_BYTES_PER_PARAM and manual parameter walk logic - Restores EP-DP divisibility validation (dp >= ep and dp % ep == 0) Stack B fixes: - OptimizerPass populates step_bytes (28 B/P) alongside state_bytes - TrainingPipelinePass reads step_bytes with legacy fallback + warning - Added mem_bw_efficiency, optimizer_comm_hidden_ms plumbing - Injects moe_ffn_hidden, layer_type_counts, vocab_size into metadata Tests: - 17 unit tests for shared primitives (test_adam_helpers.py) - 5 Stack A/B parity tests (test_adam_stack_parity.py) - 1 end-to-end OptimizerPass→TrainingPipelinePass test - Updated legacy fallback test to assert 2.3× underestimate Resolves 12 reviewer issues (correctness #1-#4, test gap #5-#8) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
增加 A2A 的 2-阶段分解（intra D 段 + inter L 段，DeepSpeed-MoE / NCCL 模式）
Implement Adam optimizer graph-based path (Stack B parity)
Fix 3 confirmed gaps where the simulator was ignoring configured quantization dtypes and always using BF16 bytes: 1. Weight memory bandwidth: Added _weight_itemsize() helper that reads quant_weight annotation. Fixed _mm(), _linear(), _addmm(), _bmm() to use separate itemsize for weight operands. 2. KV cache dtype: Added _kv_itemsize() helper that reads quant_kv annotation. Fixed _scaled_dot_product_attention() and _paged_attention() to use KV dtype for K,V operands. 3. Per-op activation granularity: Extended QuantConfig with attn_activation/expert_activation/shared_activation fields. Updated QuantizationPass to use activation_for_component(). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fix quantization memory bandwidth calculations in graph-based path
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.