Skip to content

perf: IDOT_TEAM=1 fuses three OpenMP regions into one team per expert… - #1397

Open
KyleSanderson wants to merge 1 commit into
JustVugg:devfrom
KyleSanderson:perf/glm-idot-team-fuse
Open

KyleSanderson wants to merge 1 commit into
JustVugg:devfrom
KyleSanderson:perf/glm-idot-team-fuse

Conversation

@KyleSanderson

@KyleSanderson KyleSanderson commented Sep 8, 2026

Copy link
Copy Markdown

Summary

perf: IDOT_TEAM=1 fuses three OpenMP regions into one team per expert forward
Eliminates 2x team barrier spin (gomp_team_barrier_wait_end + gomp_barrier_wait_end)
per expert forward. VTune measured 168.3 CPU-s of OpenMP imbalance/serial spin.
A/B test on GLM-5.2 real workload (--ram 8):
prefill: 73.17s -> 71.20s (-2.7%)
decode: 75.20s -> 62.07s (-17.5%)
No regression in expert-matmul IPC or output correctness.
Perf c2c confirmed no material false sharing (max HITM 2.15%).

Validation

  • make -C c check
  • CUDA changes were tested with make -C c cuda-test (if applicable)
  • Performance claims include hardware, commands, and repeatable measurements

Compatibility

  • The default CPU build remains dependency-free
  • No model files, generated binaries, or benchmark artifacts are included

… forward

Eliminates 2x team barrier spin (gomp_team_barrier_wait_end + gomp_barrier_wait_end)
per expert forward. VTune measured 168.3 CPU-s of OpenMP imbalance/serial spin.
A/B test on GLM-5.2 real workload (--ram 8):
  prefill: 73.17s -> 71.20s (-2.7%)
  decode:  75.20s -> 62.07s (-17.5%)
No regression in expert-matmul IPC or output correctness.
Perf c2c confirmed no material false sharing (max HITM 2.15%).
@JustVugg

Copy link
Copy Markdown
Owner

Status: every Linux and ARM job is red on this branch (13 of 26 checks), including the token-exact oracles, so it cannot be reviewed for the speed claim yet. The change itself is opt-in and looks numerically neutral (same per-element arithmetic, collapse(2) over the same loops), so the failures are probably the build or a stale base: please rebase onto dev and look at one red log. With CI green, the number I would need is an A/B on one binary (IDOT_TEAM=0 vs 1, same prompt, tok/s), since the getenv sits on a per-call path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants