perf(vl): optimize HunyuanOCR decoding

[GreatV]

Summary

• add CUDA Graph replay for batch-1 autoregressive decoding and generalize the existing target graph for both AR and DFlash verification
• keep LM-head projection, stable argmax, repetition history, and DFlash verification sampling on the GPU
• add fused CUDA kernels for BF16/F32 argmax and repetition penalty while preserving Hugging Face penalty semantics and first-token tie breaking
• fuse target MLP gate/up projections and support strided SiLU×mul inputs
• use adaptive KV-cache graph capacities with safe invalidation and recapture across pages and long contexts

Why

HunyuanOCR decoding was dominated by eager kernel launches and repeated full-vocabulary GPU-to-CPU transfers. Plain AR decoding had no CUDA Graph path, while the default repetition penalty forced synchronous CPU processing during both AR generation and DFlash verification.

Impact

On an RTX 4090 with .oar/images/layout.jpg, the layout prompt, repetition_penalty=1.08, and concurrency 1:

• AR: 182–230 tok/s → about 344 tok/s
• DFlash: 908–946 tok/s at 1.47–1.53 s → about 1171 tok/s at 1.19 s
• generated token sequences remain identical to the pre-optimization reference (d093cc7e76d0c08b)

Validation

• cargo test --workspace
• cargo test -p oar-ocr-vl --features cuda,download-binaries --lib — 107 passed
• cargo check -p oar-ocr-vl
• cargo check -p oar-ocr-vl --features cuda,download-binaries
• cargo build --release -p oar-ocr-vl --features cuda,download-binaries
• cargo fmt --all -- --check
• git diff --check
• release-mode AR and DFlash benchmarks on RTX 4090

[gemini-code-assist]

Code Review

This pull request introduces significant performance optimizations for HunyuanOCR, specifically implementing CUDA-graph-based autoregressive decoding and speculative decoding (DFlash). It adds fused CUDA kernels for repetition penalty, argmax, and MLP projections to minimize host-device synchronization overhead, and refactors KV cache management to support exact capacities. The review feedback suggests enhancing the robustness of the dflash_repetition_argmax_stage2 CUDA kernel by using a block-stride loop to handle arbitrary partition counts, and refactoring duplicated PARTITIONS_PER_ROW constants in dynamic_kv.rs into a single module-level constant.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[GreatV]

/gemini review

[gemini-code-assist]

Code Review

This pull request optimizes the HunyuanOCR model's speculative decoding (DFlash) and autoregressive decoding paths by fusing LM head projection, repetition penalty, and argmax operations directly into CUDA kernels and CUDA graphs, significantly reducing host-device synchronization overhead. Key improvements include fusing gate and up projections in MLP layers, implementing custom CUDA kernels for stable argmax and repetition penalty, introducing an incremental repetition history tracker, and enabling lazy, bucketed CUDA graph capture for the autoregressive decoder. No review comments were provided, so there is no feedback to address.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.