Motivation
GatedDeltaNetFwdOp is useful as a chunkwise/training-style forward primitive, but using it directly as an inference prefill operator has two user-visible issues.
-
The serving prefill contract is not aligned with mainstream linear-attention libraries such as FLA. In practice, users commonly have q/k/v in BTHD layout ([B, T, H, D]) and g/beta in [B, T, H], and expect (o, final_state) where o is BTHD and final_state is [B, H, DK, DV]. Reusing the current forward path requires extra layout adaptation and exposes training/backward-oriented intermediates.
-
The current forward path is not optimized for prefill. On Qwen-style Gated DeltaNet inference shapes, prefill needs a dedicated kernel decomposition and dispatch policy. The goal should be an FLA-compatible prefill interface whose performance is competitive with, and ideally faster than, FLA.
Proposal
Add a dedicated GatedDeltaNetPrefillFwdOp with:
- FLA-compatible BTHD as the main path:
q/k: [B, T, H, DK]
v/o: [B, T, H, DV]
g/beta: [B, T, H]
final_state: [B, H, DK, DV]
- Backward-compatible TileOPs head-major layout support for existing users.
- A zero-state serving prefill contract:
(q, k, v, g, beta) -> (o, final_state).
- Dedicated prefill kernels and dispatch rather than wrapping the chunkwise forward op.
- Benchmark coverage against FLA on representative Gated DeltaNet prefill shapes.
Acceptance criteria
- Public op API supports the FLA-compatible BTHD contract.
- Existing TileOPs-style layout users keep a supported path.
- Correctness tests cover BTHD vs head-major layout equivalence and final-state output.
- Benchmarks show the dedicated prefill path is faster than FLA on the targeted prefill workloads.
Motivation
GatedDeltaNetFwdOpis useful as a chunkwise/training-style forward primitive, but using it directly as an inference prefill operator has two user-visible issues.The serving prefill contract is not aligned with mainstream linear-attention libraries such as FLA. In practice, users commonly have
q/k/vin BTHD layout ([B, T, H, D]) andg/betain[B, T, H], and expect(o, final_state)whereois BTHD andfinal_stateis[B, H, DK, DV]. Reusing the current forward path requires extra layout adaptation and exposes training/backward-oriented intermediates.The current forward path is not optimized for prefill. On Qwen-style Gated DeltaNet inference shapes, prefill needs a dedicated kernel decomposition and dispatch policy. The goal should be an FLA-compatible prefill interface whose performance is competitive with, and ideally faster than, FLA.
Proposal
Add a dedicated
GatedDeltaNetPrefillFwdOpwith:q/k: [B, T, H, DK]v/o: [B, T, H, DV]g/beta: [B, T, H]final_state: [B, H, DK, DV](q, k, v, g, beta) -> (o, final_state).Acceptance criteria