perf(vl): accelerate GLM-OCR, MinerU, and PaddleOCR-VL

[GreatV]

Summary

• add batch-1 CUDA Graph decode paths for GLM-OCR, MinerU2.5, and PaddleOCR-VL, with bounded KV-cache buckets, eager fallbacks, and safe graph invalidation
• add lossless greedy MTP speculative decoding for GLM-OCR, including captured target verification and recurrent draft passes
• keep greedy selection, no-repeat masking, and diffusion confidence sampling on the GPU through shared CUDA kernels
• batch LM-head projection where possible and use FlashAttention/GQA paths without repeating KV heads
• centralize graph helpers, persistent KV-length scratch storage, stable first-tie argmax, and CUDA module loading across the VL backends

Why

After the HunyuanOCR optimization, the other VL backends still spent substantial decode time in eager kernel launches, per-row LM-head calls, CPU-side token processing, and repeated per-token CUDA allocations.

This change moves those hot paths onto reusable CUDA Graphs and shared GPU kernels while preserving eager semantics. In particular, single-token graph replay now uses non-causal attention just like the eager decoder, while multi-token speculative verification remains causal. This fixes the numerical drift that could otherwise prevent EOS on long or degenerate PaddleOCR-VL generations.

Correctness and safety

• CUDA Graphs are limited to supported batch, dtype, shape, and cache-capacity combinations; unsupported cases fall back to eager execution.
• Cache growth invalidates stale graphs before storage can be replayed.
• KV-length replay updates reuse a page-locked host buffer and persistent device tensor, avoiding a fresh tensor allocation on every token.
• Stable GPU argmax preserves the first-index tie behavior of the CPU implementation.
• Existing global and model-specific CUDA Graph escape hatches remain available.
• Graph and eager outputs were compared by token count and full-sequence FNV-1a fingerprint.

Representative results

Warm batch-1 runs on an RTX 4090:

Model/path	CUDA Graph	Eager	Output parity
PaddleOCR-VL formula regression case	350.7 ms	433.8 ms	131 tokens, identical fingerprint
GLM-OCR AR	418 ms	489 ms	158 tokens, identical fingerprint
GLM-OCR MTP	299 ms	330 ms	158 tokens, identical fingerprint
MinerU2.5 full layout pipeline	6.68 s	8.45 s	all 13 generated blocks identical

The canonical HunyuanOCR page case remains unchanged at 1,393 tokens with the same fingerprint; DFlash remains about 1.20 s.

These are representative local regression runs rather than a formal benchmark suite.

Review guide

1. cuda_kernels.rs and decoder_graph.rs: shared GPU selection/sampling and graph lifecycle primitives
2. glmocr/mtp.rs plus glmocr/model.rs: MTP proposal, verification, acceptance, and KV rollback
3. glmocr/text.rs, mineru/text.rs, and paddleocr_vl/ernie.rs: backend graph capture/replay and eager parity
4. model drivers: batched LM-head calls, GPU no-repeat handling, and diffusion sampling integration

Validation

• cargo fmt --all -- --check
• cargo clippy --all-targets --all -- -D warnings
• cargo clippy --all-targets -p oar-ocr-vl -- -D warnings
• RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps
• cargo test --all
• cargo test -p oar-ocr-vl --lib --features cuda,download-binaries — 117 passed
• cargo build --release -p oar-ocr-vl --examples --features cuda,download-binaries
• git diff --check

[gemini-code-assist]

Code Review

This pull request introduces shared CUDA kernels (stable argmax, sparse token masking, and sampling with confidence) and decoder CUDA-graph plumbing to optimize single-token decoding across multiple VLM backends, including GLM-OCR, HunyuanOCR, MinerU, MinerU-Diffusion, and PaddleOCR-VL. It also integrates a Multi-Token Predictor (MTP) for speculative decoding in GLM-OCR. The review feedback highlights several potential underflow vulnerabilities across the codebase where subtracting from sequence lengths or token counts (e.g., seq_len - 1, prompt_len - 1, max_seq_len - 1) could cause panics in debug mode if the values are zero, and suggests adding appropriate validation guards.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[GreatV]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: a6c4e2be66

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".