Stop creating, and stop coalescing, divergent L0C accumulator chains

docs/en/dev/ir/05-operators.md

@@ -196,6 +196,16 @@ Two limitations, both diagnosed rather than silently dropped:
   accumulate is therefore the author's responsibility, exactly as an oversized
   unpredicated `tile.matmul_acc` already is.
 
+`AutoTileMatmulL0` *emits* this form: the K-loop it builds for a plain
+`tile.matmul` is a single `tile.matmul_acc` predicated on `init_cond=(ko == 0)`,
+not an `if ko == 0` over a fresh `tile.matmul` and an in-place accumulate. That
+matters for more than tidiness. The branchy form has two producers of one logical
+value on two different L0C buffers, and every pass downstream must then agree on
+which buffer the phi lives in — with no `Acc`->`Acc` copy available when they
+disagree. A predicated accumulate is in place on both paths, so one buffer is the
+only possibility. `tile.matmul_bias` has no `init_cond` operand, so the bias form
+still branches: its first K step must apply the bias exactly once.
+
 At the tile layer, `tile.batch_matmul` provides batched semantics for
 `TileType` operands. It accepts rank >= 2 tiles, broadcasts the leading batch
 dimensions, and keeps the same operand-only interface style as `tile.matmul`.

docs/en/dev/passes/15-auto_tile_matmul_l0.md

@@ -50,7 +50,7 @@ For each `tile.matmul`, `tile.matmul_acc`, or `tile.matmul_bias` in an InCore-ty
    - Sub-byte dtypes (cube path doesn't support them) — `PH-AT-003`.
    - `ChooseL0Tile` rejects the configuration — `PH-AT-005`.
 5. **Build the K-loop** (per output sub-tile — the whole output when K-only, or each `[m, n]` sub-tile when M/N tiling):
-   - `tile.matmul` — iter-arg init is an Acc-resident `tile.create([m, n], dtype, target_memory=Acc)` placeholder; the loop body branches on `ko == 0` between `tile.matmul` (fresh Acc) and `tile.matmul_acc` (accumulating into the iter-arg). The `IfStmt` materializes a phi return_var that the outer yield carries back to the iter-arg.
+   - `tile.matmul` — iter-arg init is an Acc-resident `tile.create([m, n], dtype, target_memory=Acc)` placeholder; the loop body is a single `tile.matmul_acc` predicated on `init_cond=(ko == 0)`, so the first K step overwrites the accumulator and later steps accumulate into it. The backend expands the predicate into the `pto.tmatmul` / `pto.tmatmul.acc` pair, so the selection costs no IR-level control flow. This keeps the body in place on both paths and therefore on one L0C buffer — a branch between a *fresh* `tile.matmul` and an in-place accumulate would put one logical value on two Acc buffers, which nothing downstream can reconcile (there is no `Acc`→`Acc` copy).
    - `tile.matmul_acc` — iter-arg init is the caller's accumulator directly (its type already matches the per-iter `tile.matmul_acc` output); every iteration is uniform `tile.matmul_acc`, so no if-else.
    - `tile.matmul_bias` — uses the same fresh-accumulator loop as `tile.matmul`, but the first K block is `tile.matmul_bias` and every later block is `tile.matmul_acc`, so bias is added exactly once. For N tiling, the matching window is reconstructed as an independent tensor→Mat `tile.load`, then transferred with `tile.move` to Bias (`pto.tload` + `pto.tmov`). A zero-copy one-row Mat `tile.slice` is not used because its boxed `pto.subview` is not PTOAS-legal.
    - Per-iter operand extracts use `tile.extract(src, idx_row, idx_col, [shape], target_memory=Left|Right)` — the SSA-form fusion of the older `tile.slice` (Mat-resident result) + `tile.mov` (Mat→Left/Right) pair. This eliminates the intermediate Mat-resident slice tile and lowers to `pto.textract` rather than `pto.subview`, sidestepping the latter's `valid_row` codegen mismatch. For an output sub-tile at origin `(mi, ni)` the extracts slice `lhs[mi:mi+m, ko:ko+k]` and `rhs[ko:ko+k, ni:ni+n]`; the K-only case is `mi == ni == 0`, `m == M`, `n == N`.
@@ -167,13 +167,8 @@ class After:
         for ko, (c_iter,) in pl.pipeline(0, 256, 64, init_values=(c_l0_init,), stage=2):
             sa = pl.tile.extract(a_mat, 0, ko, [128, 64], target_memory=Left)
             sb = pl.tile.extract(b_mat, ko, 0, [64, 128], target_memory=Right)
-            if ko == 0:
-                c_first = pl.tile.matmul(sa, sb)
-                c_phi = pl.yield_(c_first)
-            else:
-                c_acc = pl.tile.matmul_acc(c_iter, sa, sb)
-                c_phi = pl.yield_(c_acc)
-            c = pl.yield_(c_phi)
+            c_acc = pl.tile.matmul_acc(c_iter, sa, sb, init_cond=(ko == 0))
+            c = pl.yield_(c_acc)
         # c (the yield-LHS) holds the accumulated Acc-typed result.
         ...
 ```
@@ -208,13 +203,8 @@ c_t1_init = pl.tile.create([256, 256], dtype=pl.FP32, target_memory=Acc)
 for ko, (c_iter,) in pl.pipeline(0, 512, 32, init_values=(c_t1_init,), stage=2):
     sa = pl.tile.extract(lhs_mat, 256, ko, [256, 32], target_memory=Left)
     sb = pl.tile.extract(rhs_mat, ko, 0, [32, 256], target_memory=Right)
-    if ko == 0:
-        c_first = pl.tile.matmul(sa, sb)
-        c_phi = pl.yield_(c_first)
-    else:
-        c_acc = pl.tile.matmul_acc(c_iter, sa, sb)
-        c_phi = pl.yield_(c_acc)
-    c_t1 = pl.yield_(c_phi)
+    c_acc = pl.tile.matmul_acc(c_iter, sa, sb, init_cond=(ko == 0))
+    c_t1 = pl.yield_(c_acc)
 out_t1 = pl.store(c_t1, [256, 0], out_t0)  # store sub-tile to out[256:512, 0:256]
 ```
 

docs/en/dev/passes/33-memory_reuse.md

@@ -59,13 +59,12 @@ program_optimized = reuse_pass(program)
 1. **Lifetime Analysis**: Walk the full IR tree (including nested control flow bodies) to compute variable lifetimes via def-use analysis. Variables defined outside a loop but used inside have their lifetime extended to the end of the loop (loop-aware extension)
 2. **Interference Check**: Identify variables with overlapping lifetimes
 3. **MemRef Sharing** (global first-fit-decreasing packing, `IdentifyReuseOpportunities`): Within each memory space, intervals are packed **largest-first**; every later interval joins the first existing buffer all of whose members it can share with (non-overlapping lifetime + hazard/no-alias safe — see `can_share`). A buffer's allocated size is fixed by its first (largest) member, so admitting a smaller member afterwards costs nothing — and a *later, larger* interval can now host an *earlier, smaller* one. (The previous definition-order greedy had a one-directional size gate `source.size >= target.size`, so two lifetime-disjoint tiles whose small one was defined first were never coalesced.) The representative each member is rebased onto is the buffer's largest member; its `tile.alloc` dominates the whole function because InitMemRef hoists every alloc to the function-body head, so a representative defined after some of its members is safe. Because the packer no longer runs in program order, every pairwise gate (hazard, no-alias) is checked in both directions.
-4. **Loop-carry re-alignment** (`AlignLoopCarriesToInitMutator`): Sharing (step 3) only retypes `AssignStmt`-defined vars (producers/init); loop-carried `iter_arg`/`return_var` nodes are excluded from the lifetime/sharing maps and keep their original MemRef. This step walks `ForStmt`s **top-down** and retypes each loop's `iter_arg`/`return_var` to its (now-reused) `initValue` MemRef, seeding `var_remap_` before recursing so a nested loop observes the corrected outer `iter_arg` as its init. Without it, a reused **nested pipelined `matmul_acc`** accumulator splits across two Acc buffers and step 5 emits invalid `acc→acc tile.move` ops that ptoas rejects on Ascend 910B ([#1352](https://github.com/hw-native-sys/pypto/issues/1352))
-5. **Accumulator if-phi coalescing** (`TopDownRetargeter::CoalesceAccumulatorIfPhis`): `LowerPipelineLoops` peels a stage-2 K-loop into `if`-phis whose live branch is an in-place `matmul_acc` (on the accumulator buffer) and whose dead `if k==0` branch is a fresh `matmul` seed on a *different* Acc buffer. Left alone, step 6 would try to reconcile them with an `acc→acc tile.move` — a second co-live L0C buffer (overflow) that ptoas also rejects (no legal Acc→Acc `tmov`). This step identifies the in-place-accumulator branch by its `reuses_input` producer and retargets the *other* branch's seed onto the accumulator buffer, so both branches share it and no move is emitted (matching `mad_acc`'s shared-`%dst` semantics). Scoped to `Acc`; the retarget is **mandatory** (a declined retarget is an `INTERNAL_CHECK`, never a move — no legal Acc→Acc move exists). It bypasses the *global* dead-at-assign liveness check (which would false-decline on the legitimate post-if phi consumer), but only after verifying the two preconditions branch exclusivity actually needs: (a) the seed producer is a `Call` lexically **inside** the branch (a pre-if value yielded through the branch runs unconditionally and would clobber the accumulator the sibling in-place branch reads), and (b) a **branch-scoped** `IsTargetDeadAtAssign` (bounded to stop at the enclosing `if`) finds no same-branch tail read of the accumulator buffer. When either fails, the phi remains uncoalesced and step 6 fails loudly rather than manufacturing unsupported Acc→Acc IR
-6. **Yield fixup**: Fix MemRef mismatches in control flow return variables. Acc→Acc copies are not legal, so a residual mismatched Acc carry is an internal error rather than a `tile.move`:
+4. **Loop-carry re-alignment** (`AlignLoopCarriesToInitMutator`): Sharing (step 3) only retypes `AssignStmt`-defined vars (producers/init); loop-carried `iter_arg`/`return_var` nodes are excluded from the lifetime/sharing maps and keep their original MemRef. This step walks `ForStmt`s **top-down** and retypes each loop's `iter_arg`/`return_var` to its (now-reused) `initValue` MemRef, seeding `var_remap_` before recursing so a nested loop observes the corrected outer `iter_arg` as its init. Without it, a reused **nested pipelined `matmul_acc`** accumulator splits across two Acc buffers and step 5 rejects the carry as unreconcilable ([#1352](https://github.com/hw-native-sys/pypto/issues/1352))
+5. **Yield fixup**: Fix MemRef mismatches in control flow return variables. There is no Acc→Acc copy on any target (nothing reads L0C but the FIXPIPE drain, whose destinations are L1 and GM), so a mismatched Acc carry is reported to the author — naming both buffers — rather than reconciled with a `tile.move`:
    - **ForStmt**: Ensure all 4 loop-carry variables (initValue, iter_arg, yield value, return_var) share the same MemRef. Inserts `tile.move` before yield if MemRefs differ
    - **IfStmt**: Patch return_vars to match yield value's MemRef
-7. **Identity-copy buffer normalization** (`NormalizeIdentityCopyBuffersMutator`): after step 5 retargets an accumulator if-phi, a downstream bare-`Var` SSA identity copy of the (now-moved) return_var can still carry the pre-coalesce buffer (e.g. `c: …mem_acc_17 = c_phi` after `c_phi` moved to `mem_acc_5`). An `x = y` copy (value a bare `Var`, not a `Call`) is a pure rename and must alias `y`'s buffer, so this single forward pass retypes such a copy's LHS to the RHS's MemRef and substitutes the LHS's downstream uses. No-op when no mismatch exists
-8. **Remove redundant allocs**: Collect all MemRefs still referenced by TileType variables, then remove `tile.alloc` statements whose MemRef is no longer in use
+6. **Identity-copy buffer normalization** (`NormalizeIdentityCopyBuffersMutator`): when an earlier step retargets a value, a downstream bare-`Var` SSA identity copy of it can still carry the pre-retarget buffer (e.g. `c: …mem_acc_17 = c_phi` after `c_phi` moved to `mem_acc_5`). An `x = y` copy (value a bare `Var`, not a `Call`) is a pure rename and must alias `y`'s buffer, so this single forward pass retypes such a copy's LHS to the RHS's MemRef and substitutes the LHS's downstream uses. No-op when no mismatch exists
+7. **Remove redundant allocs**: Collect all MemRefs still referenced by TileType variables, then remove `tile.alloc` statements whose MemRef is no longer in use
 
 **Reuse conditions**:
 
@@ -247,9 +246,8 @@ Pass MemoryReuse();
 - `ComputeLifetimes` builds MemRef sharing groups and lifetime intervals
 - `IdentifyReuseOpportunities` finds reuse candidates
 - `ApplyMemRefSharing` updates MemRef pointers via `MemRefSharingMutator`
-- `TopDownRetargeter::CoalesceAccumulatorIfPhis` coalesces peeled loop-carried accumulator `if`-phis by retargeting the dead-branch seed onto the in-place accumulator buffer, so `YieldFixupMutator` never emits an illegal `acc→acc tile.move` (see Algorithm step 5)
-- `YieldFixupMutator` fixes ForStmt/IfStmt yield/return_var MemRef mismatches after reuse (inserts `tile.move` when legal; rejects residual Acc→Acc mismatches)
-- `NormalizeIdentityCopyBuffersMutator` reconciles bare-`Var` SSA identity copies whose LHS/RHS buffers diverged after accumulator if-phi coalescing (see Algorithm step 7)
+- `YieldFixupMutator` fixes ForStmt/IfStmt yield/return_var MemRef mismatches after reuse (inserts `tile.move` when legal; reports Acc→Acc mismatches, naming both buffers)
+- `NormalizeIdentityCopyBuffersMutator` reconciles bare-`Var` SSA identity copies whose LHS/RHS buffers diverged after an earlier retarget (see Algorithm step 6)
 - `UsedMemRefCollector` gathers still-referenced MemRef pointers after sharing
 - `RemoveUnusedAllocStatements` filters out redundant `tile.alloc` statements from `SeqStmts`
 

docs/zh/dev/ir/05-operators.md

@@ -179,6 +179,14 @@ init 操作数。由于 `matmul_acc` 是原地操作（`set_output_reuses_input(
   split-K 所期望的。超尺寸的带谓词累加因而由编写者负责，这与超尺寸的无谓词
   `tile.matmul_acc` 现有行为一致。
 
+`AutoTileMatmulL0` 会*生成*这种形式：它为普通 `tile.matmul` 构造的 K 循环体是一条以
+`init_cond=(ko == 0)` 为谓词的 `tile.matmul_acc`，而不是在全新 `tile.matmul` 与原地累加
+之间做 `if ko == 0` 分支。这不只是写法上的整洁：分支形式让同一个逻辑值有两个生产者、
+落在两块不同的 L0C 缓冲上，之后每一个 pass 都必须就 phi 属于哪一块达成一致 —— 而一旦
+不一致，并不存在可用于协调的 `Acc`->`Acc` 拷贝。带谓词的累加在两条路径上都是原地写，
+因此只可能有一块缓冲。`tile.matmul_bias` 没有 `init_cond` 操作数，所以 bias 形式仍然分支：
+它的第一个 K 步必须恰好施加一次 bias。
+
 在 tile 层，`tile.batch_matmul` 为 `TileType` 操作数提供批量语义。它接受 rank >= 2 的
 tile，广播前导批量维度，并保持与 `tile.matmul` 相同的纯操作数接口风格。如果批量操作数
 需要转置语义，可以通过两种等价方式表达：在输入上显式使用 `tile.transpose(...)`，或在

docs/zh/dev/passes/15-auto_tile_matmul_l0.md

@@ -50,7 +50,7 @@ program_tiled = l0_tile_pass(program)
    - 子字节 dtype（cube path 不支持）—— `PH-AT-003`。
    - `ChooseL0Tile` 拒绝该配置 —— `PH-AT-005`。
 5. **构造 K-loop**（针对一个输出子块——K 切分时即整个输出，M/N 切分时为每个 `[m, n]` 子块）：
-   - `tile.matmul` —— iter-arg 初值为 Acc-resident 的 `tile.create([m, n], dtype, target_memory=Acc)` 占位；循环体用 `IfStmt` 在 `ko == 0` 时走 `tile.matmul`（产生新的 Acc），其它迭代走 `tile.matmul_acc`（向 iter-arg 上累加）。`IfStmt` 物化一个 phi 形式的 `return_var`，由外层 yield 写回 iter-arg。
+   - `tile.matmul` —— iter-arg 初值为 Acc-resident 的 `tile.create([m, n], dtype, target_memory=Acc)` 占位；循环体是一条以 `init_cond=(ko == 0)` 为谓词的 `tile.matmul_acc`：第一个 K 步覆盖写累加器，其余迭代在其上累加。后端把该谓词展开为 `pto.tmatmul` / `pto.tmatmul.acc` 二选一，因此这一选择不产生任何 IR 层面的控制流。这样循环体在两条路径上都是原地写、始终只用一块 L0C；若改为在全新 `tile.matmul` 与原地累加之间分支，同一个逻辑值就会落在两块 Acc 缓冲上，而下游无法协调（不存在 `Acc`→`Acc` 拷贝）。
    - `tile.matmul_acc` —— iter-arg 初值就是调用方传入的累加器（其类型已经与每次迭代的 `tile.matmul_acc` 输出一致）；每次迭代统一是 `tile.matmul_acc`，无需 if-else。
    - `tile.matmul_bias` —— 使用与 `tile.matmul` 相同的新累加器循环，但第一个 K block 使用 `tile.matmul_bias`，之后都使用 `tile.matmul_acc`，因此 bias 恰好只加一次。N 切分时，对应窗口从原始 tensor 重新 `tile.load` 到独立 Mat tile，再通过 `tile.move` 传到 Bias（`pto.tload` + `pto.tmov`）。不会使用单行 Mat `tile.slice`，因为其 boxed `pto.subview` 不满足 PTOAS 合法性。
    - 每次迭代的操作数抽取使用 `tile.extract(src, idx_row, idx_col, [shape], target_memory=Left|Right)` —— 这是旧版 `tile.slice`（Mat-resident 中间 tile）+ `tile.mov`（Mat→Left/Right）的 SSA 化合并。这样既消除了 Mat-resident 中间 slice tile，也使得 lower 后是 `pto.textract` 而不是 `pto.subview`，从而绕开后者的 `valid_row` codegen 不一致问题。对于原点为 `(mi, ni)` 的输出子块，抽取的是 `lhs[mi:mi+m, ko:ko+k]` 与 `rhs[ko:ko+k, ni:ni+n]`；K 切分情形即 `mi == ni == 0`、`m == M`、`n == N`。
@@ -155,13 +155,8 @@ class After:
         for ko, (c_iter,) in pl.pipeline(0, 256, 64, init_values=(c_l0_init,), stage=2):
             sa = pl.tile.extract(a_mat, 0, ko, [128, 64], target_memory=Left)
             sb = pl.tile.extract(b_mat, ko, 0, [64, 128], target_memory=Right)
-            if ko == 0:
-                c_first = pl.tile.matmul(sa, sb)
-                c_phi = pl.yield_(c_first)
-            else:
-                c_acc = pl.tile.matmul_acc(c_iter, sa, sb)
-                c_phi = pl.yield_(c_acc)
-            c = pl.yield_(c_phi)
+            c_acc = pl.tile.matmul_acc(c_iter, sa, sb, init_cond=(ko == 0))
+            c = pl.yield_(c_acc)
         # c（即 yield-LHS）持有累加得到的 Acc 类型结果。
         ...
 ```
@@ -196,13 +191,8 @@ c_t1_init = pl.tile.create([256, 256], dtype=pl.FP32, target_memory=Acc)
 for ko, (c_iter,) in pl.pipeline(0, 512, 32, init_values=(c_t1_init,), stage=2):
     sa = pl.tile.extract(lhs_mat, 256, ko, [256, 32], target_memory=Left)
     sb = pl.tile.extract(rhs_mat, ko, 0, [32, 256], target_memory=Right)
-    if ko == 0:
-        c_first = pl.tile.matmul(sa, sb)
-        c_phi = pl.yield_(c_first)
-    else:
-        c_acc = pl.tile.matmul_acc(c_iter, sa, sb)
-        c_phi = pl.yield_(c_acc)
-    c_t1 = pl.yield_(c_phi)
+    c_acc = pl.tile.matmul_acc(c_iter, sa, sb, init_cond=(ko == 0))
+    c_t1 = pl.yield_(c_acc)
 out_t1 = pl.store(c_t1, [256, 0], out_t0)  # 子块 store 到 out[256:512, 0:256]
 ```