[RFC] Staged PUT bandwidth optimization for GPUNetIO

[foraxe]

Summary

This RFC proposes a small, reviewable sequence for improving GPU-initiated PUT throughput without
mixing independent correctness and performance changes.

current PUT path
    |
    v
fixed completion window
    |
    v
batched WQE submission
    |
    v
independent producer QPs

Each stage keeps its original behavior as the default and has its own validation boundary.

flowchart LR
    A["Completion-serialized<br/>3.66 Gbps"] -->|"fixed in-flight window"| B["5.60 Gbps"]
    B -->|"batch reservation + one doorbell"| C["159 Gbps"]
    C -->|"4 producers, shared QP"| D["310 Gbps"]
    D -->|"independent QPs"| E["404 Gbps"]

Loading

Proposed sequence

flowchart LR
    subgraph RFC["RFC #15 — Staged PUT bandwidth optimization"]
        I12["Issue #12<br/>Validation failure still exits 0"]
        P13["NVIDIA PR #13<br/>Propagate validation failure"]

        P14["NVIDIA PR #14<br/>Fixed in-flight PUT window"]
        F1["foraxe PR #1<br/>Batched WQE submission"]
        F2["foraxe PR #2<br/>Multi-QP infrastructure fixes"]
        F3["foraxe PR #3<br/>Independent producer QPs"]

        I12 -->|fixed by| P13
        P13 -.->|validation hardening;<br/>no code dependency| P14

        P14 -->|code base| F1
        F1 -->|code base| F2
        F2 -->|code base| F3
    end

Loading

Stage	Change	Evidence	Link
Correctness	Propagate server payload-validation failure	Corruption changes server rc from 0 to 1	#12, #13
1	Keep a fixed FIFO window of PUTs outstanding	+53.2% / +77.1% / +36.6% at 4 KiB / 64 KiB / 1 MiB	#14
2	Batch contiguous PUT WQEs behind one doorbell	159.222 / 392.266 / 393.016 Gbps	stacked PR
Prerequisite	Harden flat QP-list setup and expose the resolved handler	Separate reusable setup correctness	stacked PR
3	Map four producers to four independent QPs	+30.3% / +5.74% / +5.73% over four producers sharing one QP	stacked PR

Performance was measured on one SM100 GPU with ConnectX-7 using mlx5_0, RoCEv2 GID index 3, THREAD scope, and full server payload validation.

Feedback requested: is this staged structure and benchmark evidence sufficient for reviewing the
performance series?

[YconquestY]

This is brilliant but I don't think this repo is open to external contribution...

[foraxe]

Yes, I noticed that in the README.
Our main motivation was to document the optimization path and make the implementation and benchmark methodology available, even if the patches themselves are not merged upstream. I still hope some of the ideas can be useful for future GPUNetIO or DOCA work.