[mmgaggle]

Add a GPU-initiated NVMe Key-Value path to the nvme-ep endpoint: the GPU issues KV Store then Retrieve from device code via a new nvme-ep --kv-op mode, with the value landing directly in GPU memory.

• src/include/nvme-kv.h: KV command/opcode definitions
• src/endpoints/nvme-ep/nvme-ep.{h,hip}: KV Store/Retrieve handling
• src/tester/xio-cli-options.cpp: --kv-op CLI option
• docs/nvme-kv.md: design + usage notes
• examples/: KV CPU probe and GPU-over-RADOS demo harness

Motivation

nvme-ep today drives block I/O from the GPU. This adds the NVMe Key-Value Command Set (Store/Retrieve) so device code can put/get values keyed by name, with the value transferred straight into GPU-accessible memory (host or VRAM/P2PDMA) — no CPU round-trip on the data path. The motivating workload is GPU-initiated fetch of key-addressed blobs (e.g. weight shards) directly into VRAM. The feature is additive: it's a flag on the existing endpoint, and block behavior is unchanged when --kv-op is not set.

Technical Details

• Additive flag, not a new endpoint. KV reuses the entire nvme-ep machinery verbatim — admin/IO queue creation, doorbells (including the --pci-mmio-bridge path), PRP/value-buffer allocation (host and VRAM/P2PDMA via --memory-mode 8), CQ polling, and the endpoint registry/dispatch. The only deltas are the SQE encoding and reading the returned value length from the completion. driveEndpointKv() is a sibling of driveEndpointSingle(), selected by ioParams.kvOpcode.
• Wire format follows the NVMe KV Command Set spec, so it interoperates with any conformant KV controller. Opcodes are numerically equal to block Write/Read; the controller routes them as KV because the namespace CSI is Key Value (0x1). Key bytes ride in CDW2/3 + CDW14/15, key length in CDW11, value size in CDW10; Retrieve reads the true stored value length from CQE DW0. See docs/nvme-kv.md for the full field table.
• Block-only queries are gated in KV mode (KV namespaces have no block LBA format or capacity), and the KV device routine is single-thread guarded so it can't race the shared queues if launched with --batch-size > 1.
• Reference target: validated against an SPDK vfio-user kvdev_rados namespace over Ceph RADOS (examples/stage2_kv_rados_gpu.sh); any other NVMe-KV target presenting a CSI=KV namespace works unchanged from the host side.
• Known follow-ups (tracked in the doc): batched/wavefront KV (current path is serial single-thread), and KV Delete/Exist/List opcodes (defines are present; handling is not wired yet).

Test Plan

• Build: incremental hipcc build of xio-tester (cmake --build build).
• Unit tests: ctest -R "cli-options|nvme-config|nvme-helpers|nvme-hardware".
• CPU-only path validation (GPU-free): examples/kv_cpu_probe.c issues a KV Sto/nvme0 via the kernel NVMe passthrough ioctl, exercising guest enumeration ofthe KV controller and the exact KV SQE encoding without the GPU doorbell.
• End-to-end GPU run: examples/stage2_kv_rados_gpu.sh stands up the SPDK vfio-S and runs a GPU device Store + Retrieve, verifying the value lands in GPUmemory and the stored object exists in RADOS (rados ... stat).

Test Result

• Build: clean; xio-tester and test-xio-cli-options link without warnings on the changed files.
• Unit tests: test-nvme-config, test-nvme-helpers, test-xio-cli-options pass; equires hardware).
• End-to-end GPU run: verified. The guest Linux NVMe driver enumerates the KV-only controller directly (CC.CSS=IOCS, exposes /dev/nvme0, no block namespace required); GPU-initiated KV Store then Retrieve completes, the returned value length is reported from CQE emory, and the store is confirmed via rados -p kvpool -N kvns stat.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blll-requests.

[sbates130272]

@mmgaggle thanks for this awesome contribution. Give me a few days to review and get you some feedback!

[sbates130272]

@john00003 can I ask that you take this branch from @mmgaggle for a spin on your VM setup? And provide feedback on any steps that did not work for you or were not clear?

[Copilot]

Pull request overview

Adds an NVMe Key-Value (KV) command path (Store/Retrieve) to the existing nvme-ep endpoint so GPU device code can issue KV ops and DMA values directly into GPU-accessible memory, plus CLI/docs/examples to exercise the feature.

Changes:

• Add KV SQE encoding helpers/opcodes/status codes (nvme-kv.h) and wire KV fields into nvme-ep configs.
• Implement a serial single-thread GPU KV driver path (driveEndpointKv) and gate block-only queries when KV mode is enabled.
• Add --kv-op/--key/--value-size CLI options and provide docs + example scripts/programs for CPU/GPU validation.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/tester/xio-cli-options.cpp	Adds CLI flags for enabling KV mode and setting key/value size.
src/include/nvme-kv.h	New header defining KV opcodes/status and kvSqeSetup() SQE field encoder.
src/endpoints/nvme-ep/nvme-ep.hip	Implements GPU KV path, dispatch, and host-side KV config handling.
src/endpoints/nvme-ep/nvme-ep.h	Plumbs KV fields into nvmeIoParams and host config struct; includes KV header.
examples/stage2_kv_rados_gpu.sh	End-to-end GPU KV Store/Retrieve demo against SPDK kvdev_rados via vfio-user.
examples/probe_kv_cpu.sh	GPU-free guest validation harness using CPU passthrough KV probe.
examples/kv_cpu_probe.c	CPU passthrough KV Store/Retrieve probe program to validate SQE encoding.
examples/create_rbd.c	Helper example to create/write an RBD image (ancillary tooling).
docs/nvme-kv.md	Design/usage notes for KV mode and on-the-wire field mapping.

Comments suppressed due to low confidence (1)

src/endpoints/nvme-ep/nvme-ep.hip:1573

• In KV mode, the batch buffer sizing check still uses eff_batch * (lbasPerIo*lbaSize). The KV device path is serial and always uses batch index 0, so this check can both (1) under-estimate the needed bytes if lbasPerIo/lbaSize don't reflect kvValueLen, and (2) over-reject configs by multiplying by eff_batch even though KV doesn't consume per-batch slices. It should size/validate against kvValueLen with an effective batch of 1 in KV mode.

    uint32_t eff_batch = ioParams.batchSize;
    if (eff_batch == 0) {
      eff_batch = ioParams.infiniteMode
                    ? (uint32_t)(queue_size - 1)
                    : (uint32_t)(ioParams.readIo + ioParams.writeIo);
    }
    if (eff_batch > (uint32_t)(queue_size - 1)) {
      eff_batch = queue_size - 1;
    }
    uint64_t xfer = (uint64_t)ioParams.lbasPerIo * ioParams.lbaSize;
    uint64_t needed = (uint64_t)eff_batch * xfer;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[john00003]

We need to add the nvme-kv.h header to the installed files in CMakeLists.txt:

rocm-xio/CMakeLists.txt

Lines 614 to 622 in d9e97c5

	install(FILES
	${INCLUDE_DIR}/xio.h
	${INCLUDE_DIR}/xio-endpoint-core.h
	${INCLUDE_DIR}/xio-export.h
	${INCLUDE_DIR}/rocm-xio-uapi.h
	${INCLUDE_DIR}/xio-endpoint-registry.h
	DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/rocm-xio
	COMPONENT rocm-xio
	)

Otherwise Test #112: example-nvme-ep-info fails.

Other than that, tests are passing and it compiles fine. I'll try the E2E tests now.

[sbates130272]

@mmgaggle can you review the copilot and John's comments and push an update please?

[mmgaggle]

Pushed 0aef15c addressing the Copilot review and @john00003's feedback.

Reviewer fixes

• @john00003 — nvme-kv.h is now added to the installed headers in CMakeLists.txt, so Test nvme-ep: multi-wavefront batching and multi-queue support #112 example-nvme-ep-info passes (verified locally).
• Copilot — KV op direction is validated against the io counts (store ⇒ --write-io, retrieve ⇒ --read-io), so the needed value buffer is always allocated.
• Copilot — the host PRP-list pool is sized from kvValueLen in KV mode, fixing a null PRP list / device fault for multi-page values.
• Copilot — added CLI unit tests for --kv-op/--key/--keys/--value-size.
• Copilot — example hardening: kv_cpu_probe rejects keys outside 1..16 bytes and null-checks aligned_alloc; create_rbd validates argc with a usage message.

Also in this update: wavefront/batched KV (the follow-up noted in docs/nvme-kv.md)

• driveEndpointKvWavefront() — a cooperative multi-key Store/Retrieve path: threads 1..B each encode one key's SQE into its own value-buffer slot, thread 0 rings the doorbell once and polls the B completions. Selected by --keys <manifest> + --batch-size > 1; the serial path still drives --batch-size 1.
• examples/stage3_kv_wavefront_gpu.sh runs it end-to-end against the SPDK kvdev_rados target (batched Store + Retrieve, per-key rados stat verify).
• No SPDK target changes were needed: I audited our kvdev (mem + rados) — the NVMf KV handler is async with per-command request context, and the rados backend uses librados aio, so concurrent batched in-flight commands are handled correctly.

Build is clean; 112/112 runnable ctests pass (hardware-gated tests skipped). End-to-end on real NVMe-KV hardware via stage3 is the remaining validation step.

[sbates130272]

@mmgaggle looks good. Some comments. Also are your examples wired into CTests? The main thing would be if we can auto-pull nvme-kv.h from somewhere rather than maintain a copy. That would be lovely.

Also did you not have a fix for a __threadfence_system issue? Did you want to do that as a simpler and seperate PR?

[sbates130272]

This might be better off in scripts.

[mmgaggle]

Done in 59ea56f — moved to scripts/test/probe_kv_cpu.sh (and fixed the kv_cpu_probe.c path it copies, now examples/nvme-kv/).

[sbates130272]

Can we pull this into nvme-ep-generated? Is there a public location we can curl this in from like we do for the other nvme header files?

[mmgaggle]

Good idea, and the standard defs are upstreamable. The KV opcodes/structs are the public NVMe Key-Value Command Set spec and are already in mainline spdk/spdk (include/spdk/nvme_spec.h — enum spdk_nvme_kv_opcode, spdk_nvme_kv_ns_data, plus a lib/nvme/nvme_kv.c client) from the NVIDIA KV-client contribution. So two options:

1. Keep nvme-kv.h vendored (current). It's small and spec-stable, and it carries logic a fetch can't give us: the kvSqeSetup() encoder (key packed into CDW2/3/14/15) and our vendor KV Exec opcode (0x83, ADR-0005/0014), which is not upstream in SPDK or the kernel.
2. Generate the standard defs like our other NVMe headers. A fetch-spdk-kv-headers.sh + extract step (à la fetch-nvme-headers.sh / extract-nvme-defines.sh) pulling spdk_nvme_kv_opcode/structs into nvme-ep-generated.h, leaving only kvSqeSetup() + the vendor exec bits hand-written. Note the kernel header we already fetch (linux/nvme.h) has no KV uAPI, so this would be SPDK-sourced.

Either is fine by me — which do you prefer? Happy to add the fetch script in this PR if you'd rather generate them.

[sbates130272]

@mmgaggle can you implement option 2 and fetch the headers. This avoids any mismatching in the future.

[sbates130272]

Why are these strings?

[mmgaggle]

These are the host-side CLI config fields: kvOp holds the raw --kv-op value ("store"/"retrieve"/"exec") and kvKey the --key string, mirroring the existing std::string accessPattern ("sequential"/"random") convention. They're validated on the host (IsMember for the op) and then lowered to the numeric device-side fields the GPU path actually uses — kvOpcode (uint8_t), kvKey[4] / kvKeyLen. Happy to switch kvOp to an enum if you'd prefer stronger host-side typing.

[sbates130272]

@mmgaggle can we do the ENUM please?

[sbates130272]

Is there a way we can pull this from the SPDK or Ceph code base?

[mmgaggle]

Yes for SPDK — see my detailed note on the nvme-ep.h #include thread. Short version: the standard KV opcodes/structs are in mainline spdk/spdk (enum spdk_nvme_kv_opcode in nvme_spec.h) and could be generated like our other NVMe headers. What stays hand-written either way is the kvSqeSetup() encoder and the vendor KV Exec opcode (0x83), which isn't upstream. Ceph isn't a source for these (it doesn't define the NVMe KV command opcodes). I can keep this vendored or add a fetch-spdk-kv-headers.sh — your call.

[sbates130272]

@mmgaggle as noted above let's implement a fetch please.

[sbates130272]

@mmgaggle note that @john00003 is working on a new docker directory tree and a Dockerfile for this work that includes all the forks needed to get this running. Hopefully we will have that up for review next week!

[zbennett10]

@mmgaggle @sbates130272 This is great stuff. I would love to explore the potential linking of the unique hash/namespace of a KV cache donor identified by SemBlend (https://github.com/WorldFlowAI/semblend) with a 64 byte NVMe command. Would love to work on a proof of concept here that uses some of the work we are doing with SGLang..

[sbates130272]

@mmgaggle i suggest we get this part merged. Just be sure to have some reserved fields in the command set for semantic fields we might want to add.

[sbates130272]

@zbennett10 awesome. Let's get this PR merged. Perhaps you can open an issue so we can track this. I'd love your thoughts on how to merge SemBlend semantics into a NVMe KV command set!

[mmgaggle]

One approach would to be have a vector database that stores Merkle hashes of a cache block's token prefix alongside an embedding from the prefix + cache block sequence. Then you could do a top-k search with an embedding from a new prompt to find one that is closest match. Then instead of initiating a receive for the real Markel hash, you'd use the close-to hash. If there is a way to extract embeddings from the KV data itself then that would be cool too. You potentially extract the embeddings and build a vector index by chaining store/execute/receive calls together, but it could also just be an adjacent vectordb of some sort.

[zbennett10]

@sbates130272 opened a tracking issue here: #182

My short take, just thinking through this... I would keep the NVMe-KV layer storage-oriented and put the SemBlend semantics in the metadata/manifest above it, at least for the first pass.

[zbennett10]

One approach would to be have a vector database that stores Merkle hashes of a cache block's token prefix alongside an embedding from the prefix + cache block sequence. Then you could do a top-k search with an embedding from a new prompt to find one that is closest match. Then instead of initiating a receive for the real Markel hash, you'd use the close-to hash. If there is a way to extract embeddings from the KV data itself then that would be cool too. You potentially extract the embeddings and build a vector index by chaining store/execute/receive calls together, but it could also just be an adjacent vectordb of some sort.

@mmgaggle I think this is worth exploring from an answer quality perspective - that's where we have to vet it. We have proven that you can do this (different approach, think cuVS CAGRA for vector search in the GPU, or HNSW on CPU process, local to the inference engine) and find good quality matches.

Other things to think about:

• KV tensors are model-specific.
• They are layer/head/position-conditioned.
• They encode context, not just local token meaning.
• Different RoPE positions change K.
• Raw KV is large and expensive to index.

SemBlend does work to help hand inference engines a plan that helps handle a lot of these considerations

[sbates130272]

@zbennett10 @mmgaggle can ee move this discussion to the issue for SemBlend so we don't lose it or conflate it with this PR.

[sbates130272]

@copilot can you fix the CI check failures on this branch?

[sbates130272]

@mmgaggle I think the final thing is the fetch from SPDK for the nvme-kv header files. Can we also rebase to clean up the total number of commits to just 3-4? I can then give it one last review before merging and then we can add in the docker PR from @john00003!

[sbates130272]

@mmgaggle some more issues my AI agent noticed on this branch:

What's Missing / Needs Attention Before Merge

1. ShellCheck warnings need fixing — the style guide requires shell scripts to be ShellCheck-clean with no issues. There are 3 SC2086 (unquoted variable) warnings across stage2_kv_rados_gpu.sh and stage3_kv_wavefront_gpu.sh. These are real (word-split risk on PCI BDF strings and the $KEYS manifest). Would you like me to fix them?

2. probe_kv_cpu.sh uses set +u instead of set -u — this looks like a copy-paste mistake (it disables unbound variable checking rather than enabling it). All other scripts in this repo use set -euo pipefail patterns.

3. GPU BDF is hardcoded in both stage2/stage3 scripts — GPU=0000:bd:00.0 is your specific machine. The style guide issue I flagged earlier still applies; these scripts won't work for anyone else without editing. Worth adding GPU="${GPU:-0000:bd:00.0}" pattern like the pool/ns vars use.

4. test-xio-cli-options won't run in CI without CLI11 — the test is gated behind BUILD_CLIENTS, which defaults off when CLI11 is absent. If CI doesn't install libcli11-dev, the new KV CLI option tests silently don't run. Worth checking the CI pipeline installs it, or decoupling the unit test from BUILD_CLIENTS.

5. nvme-kv.h public API has no Doxygen — the style guide requires Doxygen markup on all public API symbols. kvSqeSetup(), kvExecSqeSetup(), and the exported enums/structs in nvme-kv.h have none. Since it's installed as a public header (CMakeLists.txt puts it in the install tree), this is a firm requirement per STYLEGUIDE.md.

6. No test-rocm-xio-uapi-layout coverage for the new ioctl — the existing test-rocm-xio-uapi-layout test verifies struct layout/sizes for UAPI structs. The new rocm_xio_get_buffer_pages_req struct should have a layout assertion there like the other UAPI structs.

[sbates130272]

@mmgaggle can you sign your commits with your pgp key?

[sbates130272]

@mmgaggle I am closing this in depreciation and it is superceeded by #205! Thanks!