[Bug] MTP CSA QR accumulation tails cross INT8/top-k boundaries and cause intermittent precision failures

[wangqin1723-max]

Background

In pypto-lib, models/deepseek_v4_flash_mtp/decode_csa.py on A2/A3 has an intermittent precision problem: kv_cache passes validation, while x_out can exceed both the allowed error ratio and the maximum relative-difference limit.

A public occurrence is available in:
https://github.com/hw-native-sys/pypto/actions/runs/33036547578/job/98400778713?pr=2421

The failure is probabilistic across randomly generated fixtures. Reproduce by repeatedly running:

for run in $(seq 1 20); do
    python models/deepseek_v4_flash_mtp/decode_csa.py -p a2a3 -d 0 || break
done

Repro sources at pypto-lib commit 7ac8e24c107f3599f76ee3b30e18ac0ed7c9416a:

• models/deepseek_v4_flash_mtp/decode_csa.py
• models/deepseek_v4_flash_mtp/qkv_proj_rope.py
• models/deepseek_v4_flash_mtp/decode_indexer.py

Reproduction environment:

Component	Version
pypto-lib	7ac8e24
pypto	3e852dac (detached)
simpler	799640e6 (detached)
ptoas	0.57
pto-isa	cd4a3d3 (detached)
CANN	9.0.0

Diagnosis: pypto-lib — the device result is deterministic for a frozen fixture, and the first numerical difference is consistent with ordinary Cube/split-K FP32 accumulation-order tails. The large final error is caused by those small tails crossing pypto-lib's INT8 quantization and top-k selection boundaries. There is currently no evidence of a PyPTO, simpler, PTOAS, or PTO ISA correctness violation.

Description

The first completed numerical difference occurs in qkv_proj_rope.py at the QR projection (qr_proj_matmul).

For one frozen failing fixture with QR_OK=2:

• func14 qr_fp32: 7,156 / 8,192 FP32 values differ bitwise from the torch projection;
• maximum absolute difference: 1.43e-6;
• relative L2 difference: 2.45e-7;
• the QR INT8 tensor remains bitwise equal, but six QR scale values differ by a few FP32 ULPs;
• later BF16 and query-INT8 boundaries produce five off-by-one values;
• token 1 changes four top-k members;
• the final result has 2,929 / 131,072 points over threshold, with worst relative difference 1.321.

Changing only QR_OK=2 to QR_OK=1 makes that frozen fixture pass, but it is not a general fix.

For another fixed seed, 10270709133758572702:

• QR_OK=1 deterministically fails, while QR_OK=2 passes;
• func14 qr_fp32: 7,421 / 8,192 FP32 values differ bitwise;
• maximum absolute difference: 1.91e-6;
• relative L2 difference: 3.12e-7;
• token 1 QR INT8 element [1, 482] changes from golden 40 to device 39;
• replaying func15 from the device qr_fp32 reproduces device INT8 bitwise, so func15 is faithfully propagating its input;
• token 1 top-k changes substantially;
• the final result has 3,435 / 131,072 points over threshold, with worst relative difference 1.607.

The observed amplification chain is:

legal FP32 accumulation tail
  -> QR scale / INT8 rounding boundary
  -> score perturbation
  -> top-k selection change
  -> different sparse-attention inputs
  -> large final x_out difference

QR_OK only changes the accumulation order and therefore the direction of the numerical tail. It can fix one fixture while breaking another.

Expected behavior:

• normal FP32 accumulation-order differences should not cause the CSA selection path to violate the final precision contract;
• a proposed fix should pass both fixed fixtures and fresh-seed sweeps without relaxing the final validation threshold.

Potential fix directions requiring model-side agreement:

1. stabilize the QR quantization boundary;
2. define and stabilize the score/top-k numerical contract;
3. add both opposite QR_OK fixtures as regression tests.

Changing QR_OK globally or relaxing the final threshold is not sufficient.

Isolated split-K numerical check

An independent kernel was run with the same fixed BF16 A[8,4096] and B[4096,1024], producing FP32 outputs for QR_OK=1 and QR_OK=2 without RMSNorm, quantization, score, top-k, or attention.

Against an FP64 reference:

Configuration	Max abs	Relative L2
QR_OK=1	4.17e-6	2.01e-7
QR_OK=2	1.94e-6	1.28e-7

For the standard FP32 length-4096 dot-product forward-error bound (gamma_4096 = 2.442e-4 times the absolute product sum), the observed maximum errors consume only about 0.0168% and 0.00757% of that bound. Replaying the same binary and inputs produced identical mismatch counts and values.

This isolated result strongly supports ordinary deterministic accumulation-order tails rather than obvious toolchain corruption. It does not claim formal conformance to a PyPTO/PTOAS-specific precision contract unless such a documented contract is identified.

Host Platform

Linux (aarch64)

Additional Context

Related but not duplicate: #1043, which also shows a near-tie discrete decision being flipped by small finite numerical drift, but on a different model path and boundary.