Skip to content

[Perf][Linear-Attn] Add FLA-compatible Gated DeltaNet prefill op #1595

Description

@superAngGao

Motivation

GatedDeltaNetFwdOp is useful as a chunkwise/training-style forward primitive, but using it directly as an inference prefill operator has two user-visible issues.

  1. The serving prefill contract is not aligned with mainstream linear-attention libraries such as FLA. In practice, users commonly have q/k/v in BTHD layout ([B, T, H, D]) and g/beta in [B, T, H], and expect (o, final_state) where o is BTHD and final_state is [B, H, DK, DV]. Reusing the current forward path requires extra layout adaptation and exposes training/backward-oriented intermediates.

  2. The current forward path is not optimized for prefill. On Qwen-style Gated DeltaNet inference shapes, prefill needs a dedicated kernel decomposition and dispatch policy. The goal should be an FLA-compatible prefill interface whose performance is competitive with, and ideally faster than, FLA.

Proposal

Add a dedicated GatedDeltaNetPrefillFwdOp with:

  • FLA-compatible BTHD as the main path:
    • q/k: [B, T, H, DK]
    • v/o: [B, T, H, DV]
    • g/beta: [B, T, H]
    • final_state: [B, H, DK, DV]
  • Backward-compatible TileOPs head-major layout support for existing users.
  • A zero-state serving prefill contract: (q, k, v, g, beta) -> (o, final_state).
  • Dedicated prefill kernels and dispatch rather than wrapping the chunkwise forward op.
  • Benchmark coverage against FLA on representative Gated DeltaNet prefill shapes.

Acceptance criteria

  • Public op API supports the FLA-compatible BTHD contract.
  • Existing TileOPs-style layout users keep a supported path.
  • Correctness tests cover BTHD vs head-major layout equivalence and final-state output.
  • Benchmarks show the dedicated prefill path is faster than FLA on the targeted prefill workloads.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions