Skip to content

examples(swiglu_mx_quant): CCE + VMI - #547

Open
MirkoDeVita98 wants to merge 3 commits into
mouliangyu:feature-vmifrom
MirkoDeVita98:swiglu-mx-quant-vmi
Open

examples(swiglu_mx_quant): CCE + VMI #547
MirkoDeVita98 wants to merge 3 commits into
mouliangyu:feature-vmifrom
MirkoDeVita98:swiglu-mx-quant-vmi

Conversation

@MirkoDeVita98

@MirkoDeVita98 MirkoDeVita98 commented Jul 20, 2026

Copy link
Copy Markdown

swiglu_mx_quant: CCE + VMI kernel-test, cannsim cycle comparison + CCE multi-row DMA fix

Summary

Adds a swiglu_mx_quant kernel to the kernel-test framework with both a CCE
reference
and a 1:1 PTODSL/VMI port (bf16 in, e4m3, OCP scale, swigluMode 0),
plus a cannsim-based cycle comparison. Along the way it fixes a real multi-row
correctness bug in the CCE reference.

Both backends are byte-exact vs the golden at every swept shape. On cannsim
cycles the two are close, with a crossover: VMI is faster at small shapes, CCE
faster at large ones.

What's included

  • Kernel-test harness (kernels/swiglu_mx_quant/): spec, reference (numpy
    golden — SwiGLU silu-gate + per-32-block MX-fp8 quant + a byte-exact port of the
    CCE tiling calc), runtime, tile_config, cce/ + vmi/ backends,
    cycle_metrics.
  • VMI port (vmi/backend.py, PTODSL): per-32-block MX scaling via grouped
    vcmax(group=NBLK) + vbrc(group=NBLK); e8m0/recip bit-math; vcvt f8e4m3;
    double-buffered (per-tile MTE2/MTE3 DMA overlapped with vector compute) for
    shapes with ≥2 rows/core. VEXP/VDIV op counts match CCE exactly (equal work).
  • CCE reference (cce/, lifted from a5) + its bisheng CMakeLists.

CCE correctness fix

The CCE reference produced wrong output for shapes with >1 row per core
(rows > 64): a core wrote only its first row; later rows read as zero (y 72%→54%,
scale 50%→0% as rows/core grew). Root cause: copy_ubuf_to_gm_align_v2 drops all
but the first burst when nBurst>1
on this cannsim/bisheng target — hitting both
the scale and data DMAs. Fix: DmaUb2GmScale and DmaUb2GmY now issue one
nBurst=1 burst per row
(UB source advances contiguously, GM dest by the row
stride). DMA-only; compute and tiling unchanged; batched tiling preserved.

Result — cannsim, bf16_e4m3_rint_ocp, all shapes now y = scale = 100%:

shape before (y / scale) after
128×512 72% / 50% 100% / 100%
256×512 59% / — 100% / 100%
512×1024 56% / — 100% / 100%
1024×2048 54% / — 100% / 100%

Fair cycle comparison

Both backends byte-exact. Metric = total kernel instruction span (per-core,
blkDim=64) as the cannsim wall-clock proxy; fresh consistent sweep:

shape CCE span VMI span faster
64×512 5627 2038 VMI 2.76×
128×512 6668 3475 VMI 1.92×
256×512 6302 5019 VMI 1.26×
512×1024 7589 8071 CCE 1.06×
1024×2048 11857 16083 CCE 1.36×

Crossover ~256–512: VMI wins on the small shapes (fixed per-tile overhead
dominates there and VMI's pipeline is tighter), CCE wins at scale (its vector-engine
work is very compact — e.g. RVEC span 3287 vs VMI 14115 at 1024×2048 — while VMI's
span grows with the wider MX-scale vector work).

Caveats

  • The nBurst>1 burst-drop is likely a cannsim limitation, not real hardware;
    on hardware the original single-multi-burst CCE DMA would work without the
    per-row-burst workaround (which adds some MTE3 overhead), so CCE's true hardware
    span could be lower. This comparison is cannsim-specific.
  • RVEC/VF span is not comparable across backends (CCE __simd_vf__/bisheng
    packs vector ops differently from VMI's VPTO lowering); total instruction span is
    the fair cross-backend proxy.
  • Scope: bf16 / e4m3 / OCP / swigluMode 0. fp4 (e2m1/e1m2), f16 input, and
    swigluMode 1 are not yet exercised by the VMI backend.

Reproduce

Everything runs under cannsim . Commands from
test/kernel-test/.

Env (CANN + ptoas source build):

set +u; source /usr/local/Ascend/cann-9.0.0/bin/setenv.bash   # set +u: setenv breaks under -u
export ASCEND_HOME_PATH=/usr/local/Ascend/cann-9.0.0
export ASCEND_DRIVER_PATH=/usr/local/Ascend/driver
# ptoas source build on PYTHONPATH (build/python + MLIR core python pkg + ptodsl); PTOAS_BIN -> build/tools/ptoas/ptoas
# (or use the released ptoas wheel: `ptoas` on PATH + `ptodsl` importable)
python3 -c "from ptodsl import pto; print('ptodsl ok')"

Build the CCE reference (bisheng):

cmake -S kernels/swiglu_mx_quant/cce -B kernels/swiglu_mx_quant/cce/build \
      -DASCEND_HOME_PATH="$ASCEND_HOME_PATH" -DASCEND_DRIVER_PATH="$ASCEND_DRIVER_PATH"
cmake --build kernels/swiglu_mx_quant/cce/build --target swiglu_mx_quant_cce

Correctness (byte-exact y + scale vs the numpy golden):

scripts/run_sim.sh --output sim_outputs/smx/<be>_<shape> \
  run.py -- --op swiglu_mx_quant --backend <cce|vmi> \
             --workflow correctness --case bf16_e4m3_rint_ocp_<shape>
# prints:  y_match=…% scale_match=…%   (pass gate 99%)

Cases/shapes: bf16_e4m3_rint_ocp_{64x512,128x512,256x512,512x1024,1024x2048}.

Cycles + Perfetto trace:

scripts/run_sim.sh --output sim_outputs/smx/cyc_<be>_<shape> \
  run.py -- --op swiglu_mx_quant --backend <cce|vmi> \
             --workflow cycle --case bf16_e4m3_rint_ocp_<shape>
python3 kernels/swiglu_mx_quant/cycle_metrics.py    # RVEC / span
# Chrome-trace for Perfetto (ui.perfetto.dev):
#   sim_outputs/smx/cyc_<be>_<shape>/cannsim_*/report/trace_core0.json
# total instr-span = max(ts+dur) - min(ts) over all "ph":"X" events in that trace.

@MirkoDeVita98
MirkoDeVita98 force-pushed the swiglu-mx-quant-vmi branch 2 times, most recently from 237b884 to 5592ecc Compare July 21, 2026 11:53
@MirkoDeVita98

MirkoDeVita98 commented Jul 21, 2026

Copy link
Copy Markdown
Author

Traces for 1024 x 2048:

vmi:
vmi_1024x2048_trace.json

Screenshot from 2026-07-23 17-30-17

cce:

cce_1024x2048_trace.json

Screenshot from 2026-07-23 17-28-38

@mouliangyu
mouliangyu force-pushed the feature-vmi branch 3 times, most recently from 84a34b5 to 04d95b2 Compare July 23, 2026 09:32
mirkodevita added 3 commits July 23, 2026 14:50
…omparison

Add a kernel-test harness for swiglu_mx_quant with a CCE backend and a 1:1
PTODSL/VMI port (bf16, e4m3, OCP, swigluMode 0), plus the numpy golden and a
byte-exact tiling-data port.

The VMI port is byte-exact vs the golden at every shape. With double-buffering
(per-tile MTE2/MTE3 DMA overlapped with vector compute, mirroring CCE), it is
competitive with CCE end-to-end: faster up to 256x512, within ~6% at 512x1024,
and ~11% at 1024x2048. Per-block MX scaling uses grouped vcmax(group=NBLK) +
vbrc(group=NBLK); VEXP/VDIV op counts match CCE exactly. The residual gap at
scale is CCE running the e8m0 scale/guard bit-math on the scalar pipe while VMI
uses the vector pipe. SMX_VMI_FORCE=sb|db selects the buffering mode.

Also drop two stray [[maybe_unused]] attributes on data members in
PTOToEmitC.cpp that broke the source rebuild under GCC -Werror=attributes.
The e8m0 mxscale bytes are stored in UB by the existing PK_B16 vsts CONTIGUOUSLY
(scaleBytesPerRow bytes per row, packed back-to-back). DmaUb2GmScale previously
issued a single copy_ubuf_to_gm_align_v2 with nBurst=dim0Size; on this target only
the first burst of a multi-burst UB->GM transfer lands, so every row after row 0 was
written as zero. Result: scale_eq=100%% at 1 row/core (64x512) but 50%% at 2 rows/core
(128x512, even rows ok / odd rows zero) and lower for more rows/core.

Fix (scale DMA only, PK_B16 store unchanged): copy one row per nBurst=1 burst,
advancing the UB source by scaleBytesPerRow and the GM dest by outputScaleRowBytes.

Verified in cannsim (bf16_e4m3_rint_ocp): scale_match now 100%% at 64x512, 128x512,
256x512, 512x1024, 1024x2048 (was 100/50/25/.../.. ). y_match is unchanged by this
scale-only change (e.g. 128x512 y=72.39%% before and after) and reflects a pre-existing
vexp/vdiv precision limitation in the double-buffered compute path, not the scale DMA.
DmaUb2GmY had the identical dropped-burst bug as the scale DMA: a single
copy_ubuf_to_gm_align_v2 with nBurst=dim0Size only lands the first burst on this
target, so for >1 row/core it wrote only row 0 of each core and rows 1+ read back
as zero. This (not vexp/vdiv precision) was the cause of the batched y_match gap
(128x512 y=72%%, 256x512 y=59%%, 512x1024 y=56%%, 1024x2048 y=54%%).

Fix (batched tiling preserved): issue one nBurst=1 burst per row, rowBytes valid
bytes from the contiguous outUb row (src advances by the aligned UB row size
dim1AlignSizeNow, or /2 for fp4), landing at the GM row (dst advances by the full
GM row pitch halfInput, or dim1/4 for fp4).

Verified in cannsim (bf16_e4m3_rint_ocp), BATCHED tiling, y AND scale:
  64x512, 128x512, 256x512, 512x1024, 1024x2048 -> y=100.00%%, scale=100.00%% (all).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant