Two-GPU tensor parallelism for 16 GB boards — would you consider it, or parts of it?

[ValerioDolci]

Hi, thanks for NInfer. I know the README keeps multi-GPU out of the product boundary, so I'm asking before sending anything.

Qwen3.8-27B NVFP4 doesn't fit one 16 GB board, so I built an experimental --tp 2 --devices A,B mode on top of current master (bace20dc), building on the earlier TP2 forks (wamansou, ivanov84, parallelno). It lives in a fork: https://github.com/ValerioDolci/ninfer-tp2 (Known limitations in its README).

What it covers: ordinary decoding, --spec mtp, --spec dflash2 --lm-head-draft, prefix reuse, concurrent requests, CUDA Graph decode and --vision, with bf16/int8 KV. On two RTX 5070 Ti without P2P: decode 58 t/s plain, 110 MTP3, 118 DFlash2 at short context; GSM8K 0.975-0.98 (vLLM same weights 0.98); ctest 144/150 (the 6 failures are the tp 1 real-model tests, which do not fit 16 GB).

Design points relevant to upstream:

• --tp 1 is meant to stay identical: the two-device paths are separate, and the few shared changes are listed per commit.
• Weights are sliced at load time from the existing artifacts (no new artifact format).
• 92 linear commits, grouped so they can be reviewed in pieces: independent fixes → multi-GPU primitives → split ops and shard shapes → sharded loading → two-device forward/Program/Engine → MTP → DFlash2 → tuning → Vision.

Some commits fix things that also matter on one GPU and could come first regardless of the rest:

• the KV execution row republished on every prefill step (a stale row after a forced token);
• kernel function attributes set per device instead of once per process;
• rejecting nvfp4 TMA linear_add geometries without a route, and unsupported head counts in the causal-cache dispatch.

Would you be interested in (a) those standalone fixes, (b) the two-GPU mode as an opt-in, or neither? Happy to split, rebase or rework to your conventions.