Repository navigation
ci: test nvme-ep in a VM with an emulated GPU and NVMe SSD - #202
Closed
sbates130272 wants to merge 23 commits into
Closed
sbates130272 wants to merge 23 commits into
sbates130272 wants to merge 23 commits into
Conversation
nvme-ep has had no hardware-free CI path. build-check compiles it and test-emulate exercises test-ep in emulate mode, but nothing drives real NVMe queues from __device__ code, so the submission and completion paths only ever ran on the test node. This adds a workflow that boots a QEMU guest with a rocjitsu emulated gfx1250 attached over vfio-user and an emulated NVMe controller behind the pci-mmio-bridge, then builds rocm-xio inside it and runs the nvme-ep ctest labels against that controller. The pieces: .github/actions/rocjitsu-nvme-vm/ composite action that boots the VM .github/compose/rocjitsu-nvme-vm/ the rocjitsu + QEMU compose stack scripts/test/ansible/ guest ROCm/amdgpu provisioning scripts/test/vm-provision.sh runner-side playbook driver scripts/test/vm-guest-test.sh in-guest build and test driver .github/workflows/test-vm-nvme.yml the workflow tying them together Bring-up is a composite action rather than inline steps because the same VM is wanted by rocm-ernic and anything else without hardware to test against. It is kept free of rocm-xio specifics so it can move to sbates130272/qemu-minimal, next to qemu-tool and the other compose stacks, once it has proven itself here; consumers then switch to a `uses:` reference and this directory goes away. The QEMU image must be the -sbates-fork build: pci-mmio-bridge lives on dev/stephen/pci-mmio-bridge-submit and is not upstream, and qemu-tool hard-fails if the binary does not advertise the device. That image's entrypoint.sh ignores "$@" and execs a fixed qemu-system line, so the stack overrides the entrypoint to go through qemu-tool, which is what supplies --nvme and --pci-mmio-bridge. Every image is pinned to a dated tag, collected in one env: block so a bump is a one-line reviewable change. :latest is deliberately not used: the rocjitsu pin tracks a force-pushed review branch, and a silent retag would turn a run red with no diff to point at. Guest provisioning does not reuse qemu-minimal's vm-rocjitsu.yml. That playbook still calls vfio_guest_firmware.py, which the gfx1250-vfio- compute stack removed in favour of real firmware from the driver release matching the guest's amdgpu.ko. The playbook here asks the built module what it wants via modinfo -F firmware and fails with a list of missing blobs, rather than leaving the guest to hang in gfx_v12_1_hw_init. Whether therock plus amdgpu-dkms actually ships that gfx1250 firmware inventory on resolute is untested and can only be settled by a real run. The workflow is therefore workflow_dispatch plus a weekly schedule, not a pull_request trigger: a full ROCm and DKMS provision under emulation takes the better part of an hour, and the rocjitsu pin is not yet stable. workflow_dispatch accepts skip-gpu=1 for a build-only smoke run while that settles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
test-vm-nvme was workflow_dispatch plus a weekly cron, on the reasoning that an hour-long emulated-GPU run was too expensive to block a PR on. Run it on pull requests into main instead: nvme-ep's submission and completion paths have no other hardware-free coverage, so leaving it off the PR path means regressions land and are found a week later by a cron job nobody is watching. Add a concurrency group keyed on the PR number with cancel-in-progress. At this duration a job would otherwise stack several runs deep on an actively worked branch, and only the tip is worth testing. This repo had no concurrency convention before; the cost profile is what justifies one here. Also prefix every workflow's name: with "rocm-xio - ". These workflows show up alongside other repositories' in org-wide Actions views, where a bare "build-check" says nothing about where it came from. The check names come from the job-level name:, which is untouched, so branch protection's required-status-check list is unaffected -- unless a rule was keyed on a workflow name rather than a job name, which is worth confirming against the ruleset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
The first run of test-vm-nvme failed in 49s on "Enable KVM for the runner user". The step wrote the udev rule and then exited 1 on `test -w /dev/kvm`, with no output, because every command before it is silent on success. /dev/kvm is present on GitHub-hosted runners -- rocm-ernic and qemu-minimal both use it -- but it is root-only, and the udev rule governs node *creation*. The node already exists by the time a job starts, so `udevadm trigger` does not reliably re-apply the mode. Fall back to setting it directly, and print `ls -l /dev/kvm` so the next failure says what the permissions were rather than nothing at all. Distinguish the two ways this can go wrong. A genuinely absent /dev/kvm now fails with a message and `lsmod | grep kvm`, instead of looking identical to a permissions problem. Also pass --kvm to qemu-tool explicitly. config.py defaults kvm to True and the CLI flag defaults to _UNSET, so a config file could turn acceleration off without anything in this repo changing. There is no TCG fallback anywhere in run_vm.py -- it either emits accel=kvm or does not -- so QEMU fails loudly when the device is unusable, which is what we want here: an emulated ROCm and DKMS provision under TCG would run for hours before timing out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
With KVM working the VM job got all the way through bring-up -- images
pulled, qcow2 payload extracted, compose stack up, rocjitsu healthy,
guest accepting SSH in 23s, and lspci in the guest showing the emulated
NVMe controller, the vfio-user AMD device and the pci-mmio-bridge -- then
failed seven tasks into provisioning:
The filter plugin 'community.general.json_query' failed:
You need to install "jmespath" prior to running json_query filter
sbates130272.batesste's check_platform role resolves supported platforms
with `galaxy_info.platforms | json_query('[].name')`. json_query is a
thin wrapper over jmespath, which is not a dependency of ansible-core and
is not pulled in by community.general, so it has to be installed
explicitly.
It belongs in the runner's host tooling rather than the guest playbook:
filters are evaluated on the controller during task-argument
finalization, before anything is dispatched over SSH.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
The previous commit added python3-jmespath to the VM job's apt line and the playbook failed at exactly the same task, with the same message. apt installed ansible-core 2.16.3, but the playbook ran under 2.21. The runner image ships its own ansible ahead of /usr/bin on PATH, in an environment that cannot see dist-packages, so python3-jmespath installed fine and was simply invisible to the ansible that actually ran. The only evidence of the version mismatch was the ansible-core/2.21 URL inside an unrelated interpreter-discovery warning. Use setup-python plus pip for both packages instead, which is what rocm-ernic does, and drop ansible-core from the apt line so there is only one candidate rather than two with a PATH order deciding between them. Print ansible-playbook --version in vm-provision.sh before anything depends on it. Not knowing which ansible was running is what cost a round trip here, and the answer was not otherwise in the log. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
build-check went red on a head whose diff only touched workflow YAML, with NVME_PAGE_SIZE and NVME_SC_SUCCESS undeclared, no members in nvme_sqe or nvme_cqe, and two empty enum blocks in nvme-ep-generated.h. Running the same scripts locally against the same upstream produced a correct header, so nothing had been renamed -- the download itself had come back wrong. fetch-nvme-headers.sh used curl -sS with no --fail and no check on what landed. A 404 page, a proxy error or a truncated body was written straight to linux-nvme.h, extract-nvme-defines.sh found none of its anchors, and the empty enums it emitted surfaced much later as a wall of errors in nvme-ep.h with nothing pointing back here. Fetch to a temporary file with --fail --location --retry, and publish it only after grepping for the anchor the extractor depends on. A bad fetch now fails at download time saying which symbol is missing from which ref. Pin KERNEL_REF to v6.18 rather than tracking master. The extractor greps for specific kernel symbols and renames some on the way out, so master moving is a build break with no diff to attribute it to. v6.18 generates a byte-identical header to master today, so this changes nothing about the current build. KERNEL_REF is overridable for a one-off check against a newer kernel. This script is the only place the ref is stated, and cmake's fetch-nvme-headers target runs it, so local builds and CI resolve the same headers by construction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
The previous commit made a bad header download fail loudly instead of
silently corrupting the build, and the very next run showed what had
been corrupting it:
curl: (22) The requested URL returned error: 429
curl: (22) The requested URL returned error: 429
ERROR: failed to download include/linux/nvme.h
raw.githubusercontent.com rate-limits by IP, and every CI job that
builds starts at the same moment behind the same runner egress address.
Under the old script the 429 body was written straight to linux-nvme.h,
extract-nvme-defines.sh found none of its anchors, and the empty enums
it emitted surfaced as undeclared NVME_PAGE_SIZE and missing nvme_sqe
members in nvme-ep.h -- a red build with no diff to attribute it to.
Failing loudly was the right change; it just was not yet a fix.
Use the api.github.com contents endpoint when a token is available. It
serves byte-identical content under a 5000/hr per-token budget instead
of a shared per-IP one, verified here against both endpoints. Workflows
that build now pass github.token. Without a token the script still uses
raw.githubusercontent.com, which is fine locally where there is no
burst, and says so if it gets rate-limited anyway.
Also drop --retry-delay so curl backs off exponentially. A fixed 2s
delay spent all four attempts inside the same rate-limit window, which
is why the retries in the previous commit bought nothing.
This logic is now shared in gh-fetch-lib.sh, because
fetch-rdma-headers.sh had the same defect in a worse form: curl -sL with
no --fail and "|| echo Warning", so a failed fetch left an HTML error
page on disk under a .h name and the build carried on. It has been
writing the literal "404: Not Found" into mlx/mlx5_cq.h on every build
since rdma-core v62.0 dropped providers/mlx5/cq.h. Nothing includes that
file, so this is invisible today, but an error page named like a header
is a bad thing to keep around. Those fetches stay non-fatal, as a
provider the build may not use should not break it; they just no longer
leave garbage behind.
The VM job is deliberately left alone: its build runs inside the guest,
and passing a token in there is a separate decision.
Separately, fix the gfx1250 firmware assertion in the VM playbook. It
failed during task-argument finalization with "Missing end of comment
tag": bash's ${#missing[@]} opens with "{#", which Jinja reads as a
comment tag. The task never ran, so the firmware inventory is still
unverified. Wrap the body in {% raw %} and pass the module path through
the environment so nothing in the script needs templating.
That run reached ok=72 changed=25 before this, against task 7 before the
interpreter fix, and ansible-playbook --version confirmed core 2.21.4
under the setup-python interpreter -- one interpreter, as intended.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
The firmware assertion added last commit finally ran, and failed with
all 44 blobs missing. Two of those are bugs in the assertion itself; the
rest were real, because nothing ever installed the firmware.
The playbook header claimed "we install the firmware packages and then
assert the inventory", and the workflow step is named "Provision ROCm,
amdgpu and firmware in the guest", but no such task was ever written.
Only ip_discovery.bin was placed. The assertion was guaranteed to fail
the moment it could execute.
There is no package to install. amdgpu-dkms already pulls in
amdgpu-dkms-firmware, so it was present -- but release 31.50 ships
gc_10_3_6 through gc_12_0_1 and sdma_4_4_2 through sdma_7_0_1, with no
gc_12_1_0 and no sdma_7_1_0. 31.50 is the newest release on
repo.radeon.com, and upstream linux-firmware 404s on all four gfx1250
blobs (probe validated against gc_12_0_0_imu.bin and gc_11_0_0_imu.bin,
both 200). The guest's amdgpu.ko requests firmware no public source
ships: the DKMS source knows about gfx1250, the firmware package does
not.
So do what qemu-minimal does and synthesise the stubs with
vfio_guest_firmware.py, including the uni_mes stand-in for the older MES
paths the generator does not emit. The header's claim that the
gfx1250-vfio-compute stack removed that script was right about the fact
and wrong about the conclusion: it was removed from the *image*, not
replaced by anything. Probing three tags:
rocjitsu.730bc62 vfio_guest_firmware.py present
rocjitsu.909c17f absent <- our pin
latest absent
qemu-minimal's working branch pins 730bc62 in all twelve places for
exactly this reason. Rather than downgrade the whole stack, pin
ROCJITSU_FIRMWARE_IMAGE separately: 730bc62 synthesises firmware,
909c17f still serves the device, since that is the tag the emulation has
been proven against. Note qemu-minimal's main still points at :latest
and its ansible-playbook-test is currently failing.
The two assertion bugs, both of which would have produced false
failures even once firmware existed:
* amdgpu-dkms-firmware installs under /lib/firmware/updates/amdgpu,
not /lib/firmware/amdgpu, so blobs that are present and loadable
were reported missing. Search both.
* modinfo lists firmware for every ASIC the module supports, so
matching "mes" demanded MES blobs for a dozen unrelated GPUs never
loaded for this device. 41 of the 44 entries were noise. Match
gfx1250 only.
Also add scripts/test/run-vm-nvme.sh, which is how the above was found.
It runs this workflow outside GitHub Actions on any host with docker and
a usable /dev/kvm -- a Slurm storage node, say -- and takes its image
pins out of the workflow rather than restating them, so the two cannot
drift. A storage node reaches the firmware assertion in about 20
minutes against 36 in CI, and unlike CI you can keep the guest and poke
at it. It defaults to a larger guest than CI's 4 vCPU / 8 GiB, with
--vcpus/--vmem to dial back to the CI shape when reproducing a CI
failure.
--skip-gpu now reaches provisioning as well as the guest test, via a
pass-through of extra args in vm-provision.sh, because the firmware
assertion would otherwise fail the run before the NVMe half was reached.
Verified: both image pins resolve and do not collide, shellcheck is
clean at default and CI severity, the playbook parses as YAML and Jinja,
and the assertion body passes bash -n. The generation path itself has
NOT yet been proven end to end -- a run on a storage node is in the ROCm
install as this is written, and has not reached the firmware tasks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
The firmware generation added last commit works. A storage-node run
reached "all requested gfx1250 firmware blobs present" with ok=78
changed=31 failed=0, which closes the caveat that commit ended on: the
generation path had not been proven end to end, and now it has.
The run still failed, at "no KFD agent after amdgpu load". That turns
out not to be a firmware or an emulation problem.
The guest autoloads amdgpu at boot. That happens long before this
playbook runs, so at that point amdgpu is neither blacklisted nor
supplied with firmware, and the probe fails:
[4.079] amdgpu 0000:00:06.0: initializing kernel modesetting
(IP DISCOVERY 0x1002:0x75C1 ...)
[4.256] Direct firmware load for amdgpu/psp_15_0_8_toc_1.bin
failed with error -2
[4.261] amdgpu 0000:00:06.0: Fatal error during GPU init
The probe fails but the module stays resident. So when amdgpu-probe
later runs modprobe with emu_mode=1 and friends, modprobe sees the
module already loaded, ignores the parameters and returns 0. Nothing
rebinds, nothing is logged, and the only symptom is a KFD node that
never appears -- which is why the guest dmesg collected at the end of
the run contains no amdgpu output past t=4.2s.
Blacklisting alone cannot fix this: the file is written to a guest that
has already booted, and nothing reboots it afterwards. Unload first
instead, which is correct whether or not the module happens to be
loaded. The compose log supports the diagnosis: a later boot, with the
firmware in place, gets past early_init and fails -22 instead, which is
what amdgpu does on this device without emu_mode=1.
Also check that the driver actually took the device. modprobe returns 0
when the module loads and the PCI probe fails, so a successful modprobe
says nothing about whether there is a GPU to talk to; without this the
next failure of this shape would again surface sixty pointless seconds
later as a missing KFD node rather than at the point of failure. Match
the driver's sysfs directory by glob rather than hardcoding 00:06.0,
since the BDF depends on how the compose file orders the devices.
Separately, bump the qemu fork pin to 20260916-qemu.98eca1a-vfu.8039244.
The tag suffix now names the qemu commit; previously it named the qemu
*branch*, so a rebuild could change QEMU's source and version under an
otherwise identical tag suffix -- 20260915 and 20260916 shared the
suffix pci-mmio-bridge-submit-vfu.8039244 while differing in QEMU
(11.0.2 to 11.1.1), Ubuntu (24.04.4 to 24.04.5) and qemu source commit
(cf0befc to 98eca1a). Prefer commit-suffixed tags, and say so where the
pin lives so it is not silently reverted to a branch-suffixed one.
Verified: the generated helper passes bash -n and shellcheck at warning
severity, extracted from the playbook via yaml.safe_load rather than by
slicing text. The amdgpu reload itself has NOT yet been proven: a run
carrying both changes is in flight on a storage node as this is written
and has not reached GPU bring-up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
The test VM booted the "basic" guest flavour, a bare Ubuntu, so every run
installed ROCm userspace, built amdgpu-dkms and rebuilt it after patching
-- the better part of an hour before a single test ran.
There is now a "rocjitsu" flavour built to this repo's requirements. See
ubuntu-qcow2-gen/consumers/rocm-xio-rocjitsu.md in batesste-ci-images for
the contract it publishes. It carries ROCm userspace from the therock
stream, amdgpu-dkms installed and patched, the HWE kernel, rocm-xio's
build dependencies and a blacklist on amdgpu autoload. Switch to it and
delete everything it makes redundant:
* the rocm_setup role, and with it both pre_tasks -- the apt pinning
and the stale keyring cleanup existed only because we installed ROCm
ourselves -- and the stale distro package purge
* rocm-xio's build dependencies and amdrocm-blas-dev
* the Galaxy collection install in vm-provision.sh, since no role is
left to provide, and jmespath in the workflow, since json_query came
only from that role
The playbook goes from 78 tasks to 14. What remains is exactly what the
image deliberately omits: the gfx1250 firmware and ip_discovery.bin,
amdgpu-probe, and passwordless sudo.
The KFD atomics patch and the DKMS rebuild go too, and that is the one
substantive behaviour change rather than a simplification. The driver
that loads on this image is the guest's *in-tree* amdgpu, not
amdgpu-dkms: noble's 7.0 HWE kernel is the first whose in-tree amdgpu has
GC 12.1.0 and names the gfx1250 blobs, and amdgpu-dkms is a ~24.04
package whose kcl layer cannot compile against 7.0 at all. The image
ships a patched DKMS tree built for 6.8, which is inert. So patching that
tree changes nothing about what modprobe loads, and rebuilding it costs
the time this commit is trying to recover. If KFD's PCIe-atomics gate
does fire against the emulated device, the fix has to move into the
in-tree driver -- a kfd_device.c patch plus a kernel build -- and that is
unresolved either way.
Dropping amdrocm-blas-dev is safe despite the image doc's warning that it
is headers-only and will link but fail at load: rocm-xio does not
reference BLAS anywhere in its sources or its CMake, so the package was
never needed.
The amdgpu blacklist is kept even though the image ships one, so this
playbook does not silently depend on it. That is the failure debugged in
the previous commit and it costs one file write.
Changing the driver changes the firmware inventory, which is what the
first run on this image found. In-tree 7.0 names all seven gfx1250 blobs
where amdgpu-dkms named three, so it asks for gc_12_1_0_rlc.bin, which
the generator does not emit -- it emits gc_12_1_0_rlc_1.bin, the name
amdgpu-dkms happened to ask for. That is the same suffix mismatch already
handled for uni_mes, so generalise the stand-in into one guarded loop
covering both. The stubs are header-only fixtures with sentinel payloads
rather than microcode, so which one stands in for which matters only in
that the header parses.
Note the payload grows from 3.5 GB to 6.4 GB and the qcow2 from 4.88 GB
to 17.8 GB. Irrelevant on a storage node; on a GitHub-hosted runner the
pull plus extraction is roughly 24 GB on the root volume, which should
fit but with far less margin than before.
Verified: the playbook parses as YAML with 14 tasks, the new when-guard
renders True and False correctly against both stat outcomes, and
vm-provision.sh passes bash -n and shellcheck at warning severity. A run
on a storage node reached the firmware assertion in about two minutes
against twenty, with the ROCm install gone entirely, and failed only on
the single missing blob fixed here. That fix itself has NOT yet been
proven: no run has been made carrying it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
run-vm-nvme.sh reads its four image pins out of the workflow so that a
local run and CI cannot drift apart. That is the right default and it
stays the default, but it also means swapping one image for another
requires editing the workflow, which is exactly the wrong thing to do
while working out whether a failure belongs to the image or to the code
under test.
Take each pin from the environment when it is set and fall back to the
workflow otherwise, and mark any pin that did not come from the workflow
as overridden in the banner, so a log cannot be misread as describing the
pinned configuration when it does not.
This came out of the first run on the rocjitsu guest image. Provisioning
is clean there -- ok=15 changed=8 failed=0, with the firmware assertion
reporting every requested gfx1250 blob present -- but the in-tree driver
then refuses the device:
detected ip block number 0 <common_v1_0_0> 1 <gmc_v12_0_0>
2 <ih_v7_0_0> 3 <gfx_v12_1_0>
4 <sdma_v7_1_0>
Failed to add vcn/jpeg ip block(UVD_HWIP:0x0)
Fatal error during GPU init
probe with driver amdgpu failed with error -22
amdgpu_discovery_set_mm_ip_blocks() takes its default case and returns
-EINVAL because rocjitsu does not model VCN/JPEG and the discovery table
says so. Nothing on this side can steer around it: rj-ip-discovery takes
a generation and an output path and has no VCN option, ip_block_mask is
applied when blocks are added rather than when discovery builds them, and
there is no module parameter for it. Note the bind check added two
commits ago is what turned this into a legible error at the point of
failure rather than a missing KFD node sixty seconds later.
So the question is whether that rejection is specific to the in-tree
driver or whether the emulated device cannot be brought up at all -- no
run in this branch has ever produced a KFD agent, because every earlier
attempt died on the module-already-loaded bug or was cancelled. Answering
it means running the previous commit, which still carries both the basic
image pin and the provisioning that builds amdgpu-dkms. An env override
makes that a one-line change instead of a temporary commit.
Verified: bash -n and shellcheck at warning severity are clean, and the
${VAR:-$(pin VAR)} form does not invoke pin when the variable is already
set, so an override cannot be overwritten by the fallback.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
The rocjitsu VM job never got as far as running a test: it failed in provisioning, then in CMake configure, then in the HIP compile, each time for a different reason. Fix the chain end to end, and add enough tracing and documentation that the next failure is diagnosable from the run log instead of from an interactive VM. Pin the resolute qcow2 build, not noble. Both boot 7.0.0-31-generic, but repo.radeon.com keys its apt suite off the release codename, so noble installs the ~24.04 amdgpu-dkms whose kcl layer cannot build against 7.0. That image ships a module built for 6.8 that can never load, the in-tree driver loads instead, and it rejects the emulated device outright because rocjitsu models no VCN/JPEG. Resolute gets the 26.04 package, which builds, loads, and brings up a KFD node. The ansible header is corrected to match: the DKMS module and the KFD atomics patch are both applied at image build time, so the playbook neither rebuilds nor patches. Add cmake/FindNUMA.cmake. The therock hsakmt config calls find_dependency(NUMA), which is a find_package in disguise; Debian's libnuma-dev ships numa.h and libnuma.so but never a NUMAConfig.cmake, so installing the package cannot satisfy it. The module defines both NUMA::NUMA and numa::numa as GLOBAL imported targets, because hsakmtTargets.cmake links the lowercase spelling. Install libnuma-dev in the playbook so the module has something to find. Add cmake/XIOSanitizeImportedTargets.cmake and apply it to hsakmt::hsakmt. That target's link interface leaks its manylinux build machine -- an absolute /usr/lib64/libc.so and a /__w/rockrel/... libdrm path -- and make fails with "No rule to make target '/usr/lib64/libc.so'". Drop non-existent absolute entries and leave -L flags alone. This is a no-op against ROCm packages that do not leak. Anchor the rsync excludes in both copy paths. An unanchored "build" matches at any depth and silently dropped scripts/build/, so the guest failed with a missing fetch-nvme-headers.sh that read like a CMake bug. Add waitcntAll() to sdma_device.hpp. gfx12 split the unified s_waitcnt counter into loadcnt/storecnt/kmcnt, so llvm.amdgcn.s.waitcnt has no encoding and __builtin_amdgcn_s_waitcnt(0) fails instruction selection outright. The wait sequence mirrors the one ringDoorbellFenced() already uses for gfx12 in src/include/xio.h. Plumb XIO_FORCE_PCI_MMIO_BRIDGE through the nvme-ep test wrappers. The ctests pin USE_PCI_MMIO_BRIDGE=0 in their ENVIRONMENT property for hardware, and ctest's ENVIRONMENT adds to the inherited environment rather than replacing it, so that pin beats anything exported outside. A differently named variable is the one lever that reaches these tests without editing thirty definitions. Collect QEMU trace events. qemu-tool --pci-mmio-bridge already enables pci_mmio_*, but the output went to the container's stderr and died with the stack; --trace now selects the controller side too and the events are collected with the other diagnostics, with a histogram and an I/O-queue doorbell count printed at the end of the run. Deliberately no --nvme-trace-file: this QEMU lacks the simple trace backend, so -trace file= is accepted and silently does nothing. Document the whole setup in AGENTS.md, including the traps that cost the most time here: Slurm reaps the compose stack when the job ends, so interactive debugging must happen inside one allocation; /tmp is not shared between login and compute nodes; and the bridge polls an 8 KiB shadow window in guest RAM rather than snooping the NVMe BAR. Device-touching nvme-ep tests still hang rather than pass. A backtrace shows the host in hipDeviceSynchronize() from xio::syncHipKernel(): the GPU kernel launches and makes no progress in 600 seconds, so it is stuck rather than slow. No doorbell on the tester's queue reaches QEMU and the bridge logs nothing past its startup events, with or without the bridge enabled. That is recorded in AGENTS.md as known-bad and is not addressed here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
The AGENTS.md section added in ba857c0 introduced nineteen terms aspell does not know, and spell-check failed on all of them. They are host names and flags (cx, smc, strg, vcpus, cpus, workdir), QEMU and PCIe device options (ioeventfd, dbcs, sqid), tool and project names (Slurm, rocjitsu, therock, retag, OOM, backtrace), test nouns (ctest's, ctests) and two HIP symbols named in the known-bad note (hipDeviceSynchronize, syncHipKernel). Verified by compiling .wordlist.txt into an aspell master dictionary locally and re-checking AGENTS.md against it: without these entries the run reproduces the exact CI list, with them it is clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
Image tags gained a .g<ci-commit> field, so they now read <date>.g<ci-commit>-<component-commits>. The leading date alone was never unique: the same date+component tag was re-pushed against a new digest, so a pin silently changed meaning between runs. The new field makes every rebuild a distinct tag. Move QEMU_IMAGE, ROCJITSU_IMAGE and QCOW2_IMAGE onto the new scheme in both the workflow and the compose env template, and record why the field must not be dropped. The new pins are byte-identical re-tags of the images the debug VM has been running, so this cannot change CI behaviour: rocjitsu, qemu-system-x86_64 and libvfio-user.so.0.0.1 all match by sha256, and the qcow2 image is a single layer with the same digest. ROCJITSU_FIRMWARE_IMAGE stays on the old scheme. It pins rocjitsu.730bc62 for vfio_guest_firmware.py, and 730bc62 was never rebuilt under the new scheme, so there is no tag to move it to. That pin therefore still carries the re-push risk, which the comment now spells out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
rocjitsu serves only the DMA windows a client shares mmap-ably and drops the rest with "vfu: ignoring the guest window at 0x80000000+8192". The pci-mmio-bridge shadow BAR was one of the dropped ones, because QEMU allocated it as anonymous RAM with no backing fd to hand over. Move QEMU_IMAGE to qemu.7794baa, which backs that region with a memfd and marks it shared, so the window arrives with an fd and rocjitsu maps it. Verified on the debug VM: the "ignoring the guest window" line for the bridge is gone, only the SeaBIOS ROM window at 0xfffc0000 remains, and amdgpu-probe returns 0 with KFD topology nodes 0 and 1 present. ROCJITSU_IMAGE keeps the same rocjitsu source commit, 909c17f. Only the .g<ci-commit> field moves, so this half is a re-tag and not a behaviour change. Also record what a component suffix does and does not promise. It names the *source* commit, not the build: a cmake option change such as RJ_LOG_GROUPS produces a different binary under an unchanged suffix, and four such rebuilds turned out to be byte-identical. Compare digests, not suffixes, when a rebuild is meant to change behaviour. The registry additionally carries variant-only tags and :latest, both of which move, so only the fully specified form is stable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
The rocjitsu VM suite configured the guest build as CMAKE_BUILD_TYPE=Debug, which hands the HIP compiler "-g -g" and no -O at all, so device code was built at -O0. An -O0 code object never completes a dispatch on the emulated gfx1250 that rocJITsu serves: the launch is accepted, hipDeviceSynchronize() then spins in BusyWaitSignal::WaitAcquire on a completion signal that never arrives, and this reproduces with a one-line empty kernel. -O1 and above are unaffected. This is what the suite's ctest timeouts actually were. It had nothing to do with doorbell coherence, the pci-mmio-bridge or the emulated NVMe controller -- test-doorbell-coherence deadlocked on its very first store, before any doorbell was rung, because its kernel was never running. The reason this took a while to see is that it only bites through CMake. We select amdclang++ over hipcc just above, and the hipcc wrapper quietly adds -O3; every reproducer built by hand with hipcc passed at 4096 iterations, while the CMake-built binary of the same source hung. With RelWithDebInfo, in the VM against rocJITsu and the emulated NVMe: test-doorbell-coherence deadlock -> PASSED, both variants, 4096 iterations ctest -L nvme 11/43 passing -> 23/43 The 20 that still fail are a separate problem and a different shape: they now fail in seconds on data verification rather than timing out. Signed-off-by: Stephen Bates <sbates@raithlin.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
Replace the "Known-bad state" note, which described the nvme-ep timeouts as an unexplained hang, with the actual cause, and record two things that make the VM's results easy to misread. rocJITsu's emulated gfx1250 does not correctly execute any kernel whose .amdhsa_private_segment_fixed_size is greater than zero. Such a kernel either spins forever in hipDeviceSynchronize() inside BusyWaitSignal::WaitAcquire, or returns hipSuccess having had no effect at all. Measured at 272, 1040 and 10256 bytes, all fail identically; the same kernel at size 0 returns the right answer, so it is not a threshold effect. -O0 device code always uses scratch: it spills the kernel's own arguments before doing anything else, so a one-line kernel assembles to 683 scratch instructions. CMAKE_BUILD_TYPE=Debug gives the HIP compiler "-g -g" and no -O, which is why every GPU test in the guest hung. -O2 does not rescue the endpoint. nvme_ep::gpuKernel itself has a 10320 byte private segment, from the batch timing array and the four PRP arrays held as locals in driveEndpointSingle, and every GPU-using nvme-ep test dispatches it. Also warn that xio-tester exits 0 even when every verification failed, so ctest records the run as passing. All three nvme-verify-inline-* tests that currently report Passed report zero verifications passed; across the verify label not one nvme verification has ever succeeded in the VM. A green nvme run has to be confirmed by reading the Verify Passed counter under ctest -V. Signed-off-by: Stephen Bates <sbates@raithlin.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
Every GPU test in the nvme label hung or timed out under rocJITsu. Three separate causes, none of which is a rocm-xio defect on real hardware, and all three of which have to be fixed at configure time. Scratch. driveEndpointSingle() keeps five 256-entry arrays in private memory, 10256 bytes per thread. A gfx12 runtime provisions scratch for full occupancy regardless of grid size, so that is ~336 MB against an emulated device that can back about 43 MB, and every dispatch needing scratch hangs forever. XIO_NVME_MAX_BATCH_DEPTH turns the 256 into a knob; it bounds submission batch depth, not the I/O count. The default stays 256 for silicon and the guest runner configures 32. Timeouts. `ctest --timeout` cannot help here: it only supplies a default for tests with no TIMEOUT property, and every test in this tree ends up with an explicit one. XIO_CTEST_TIMEOUT_SCALE multiplies every budget the cmake helpers set, applied through _xio_scaled_timeout() in XIOTestHelpers, XIOInstallTests and the four CMakeLists that called add_test() directly. Default 1; the guest runner uses 4, which is what the measurements support -- the worst observed inline run was 160.79 s against a 120 s base. Note that in-script timeouts are not scaled, and that scaling hides regressions: the last two "it is just a ctest timeout" episodes on this path were genuine bugs. Budgets. The six inline verify tests were given 30 s and 60 s, which was never enough even before scaling; they are now 120 s. Five of the six pass with that. nvme-verify-inline-gt8k-device-mem still exceeds 900 s with its exact ctest arguments and is not a budget problem. CI moves to an image built from the vfio-rocjitsu tree rather than the ci-images family, because stock rocJITsu does not get this suite past bring-up. Five local patches are needed; the last of them makes the CP ask a gfx12 runtime for scratch at all. The tag names the upstream rocJITsu commit and a sha256 over the patch series, since what makes the image work is the patches, not the branch. NotesForRocjitsu.md in that tree documents each one for the maintainers; when they land upstream this pin should move back to a plain ci-images tag. timeout-minutes goes 120 -> 240 to leave room for the scaled budgets, and a timeout-scale workflow_dispatch input lets a run override the scale without a commit. Also included: run-vm-nvme.sh now prefers a local image over a pull when one exists, which is the point of being able to override a pin, and the guest playbook hardens the amdgpu blacklist -- a modprobe.d file alone does not stop the initramfs, so it also sets modprobe.blacklist=amdgpu on the kernel command line and asserts both that the flag is present and that amdgpu is not already resident before probing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
sbates130272
force-pushed
the
feat/rocjitsu-ci
branch
from
September 18, 2026 05:52
c97b4f8 to
728bbd8
Compare
The AGENTS.md sections on rocJITsu scratch and the nvme false greens introduce ten terms the wordlist does not carry: symbol and identifier names quoted from ROCr and nvme-ep, a hex literal, a cmake build type, and two ordinary words aspell happens to reject. These were never checked before. GitHub skips pull_request workflows while a PR is CONFLICTING, so spell-check had not run against this branch since the AGENTS.md text landed; rebasing onto main made the branch mergeable and the job ran for the first time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
The blacklist assertion read /proc/cmdline with lookup('file', ...), and
lookups are evaluated on the Ansible controller. So it tested whichever
machine ran ansible-playbook and said nothing at all about the VM.
It passed locally by coincidence: this workstation carries
modprobe.blacklist=amdgpu itself, so the controller satisfied an
assertion meant for the guest. On a GitHub runner it does not, and the
job died in provisioning with the runner's own command line quoted back
as the guest's.
Use the ansible_cmdline fact, which is gathered from the target and is
already parsed into a dict, so the check is an exact match on the
parameter rather than a substring scan.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
ci-images now builds the rocjitsu image with the five vfio-rocjitsu patches applied, so the private image that carried them is no longer a separate lineage. The two builds of 20d4ce1c are the same binaries, not merely equivalent ones: rocjitsu and librocjitsu.so match by md5, the recorded patch diffstat is identical file for file, and the strings that only patches 0003 and 0005 introduce are present in the shipped binary. That also retires the second pin. ROCJITSU_FIRMWARE_IMAGE existed only because vfio_guest_firmware.py had been dropped from the image, which forced firmware synthesis onto a tag held back at rocjitsu.730bc62 while the device was served by another. The generator is back, so the variable now defaults to the serving image and the workflow states one tag. It stays overridable, which is how a firmware regression gets bisected against a tag known to serve the device correctly. The generator that came back is a rewrite, not the old file. It keeps the --output interface and emits a superset: the same header fixtures plus the two MES names, ip_discovery.bin and a manifest naming the set. The playbook's stand-in copies are stat-guarded, so the names it used to fabricate are simply overwritten with the same bytes, and the unsuffixed rlc blob still comes from the copy step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
vCPUs are NVMe I/O queues. The driver takes min(num_online_cpus, max_ioqpairs), so the hardcoded vcpus: "4" gave the guest a controller with four of them: nvme nvme0: 4/0/0 default/read/poll queues The qid stress tests default to STRESS_CPU=7 and derive QID=$((CPU + 1)), so all four of them target QID 8 and get ERANGE back on every iteration of every run -- which the guest dmesg, readable for the first time on run 35368507895, spells out as "qid 8 out of range (nvme0n1 has 4 hw queues)". Three more tests die on the hijack guard, which will not take --num-queues 4 from a controller with four queues to its name. Seven of the fifteen failures in that run were the guest being too small, not rocm-xio being wrong. run-vm-nvme.sh boots 16 vCPUs, which is why none of this reproduced by hand. Derive the size from the runner and floor it at eight, the way rocm-ernic's system-tests lanes derive theirs and floor at four for IONIC_EQ_COUNT_MIN. A larger runner class then helps without another edit here. Oversubscribing host cores is the cheaper mistake: rocjitsu interprets every dispatch regardless, so the guest is not host-CPU-bound in the way the vCPU count suggests. Bound the compose log collection while here. rocjitsu is built with RJ_LOG_GROUPS=CP and util/log.h reads that into a constexpr, so there is no runtime gate on it: the unbounded collection wrote a 19.3 GB artifact in 14m53s on this run, and on the run before it was still going 74 minutes in when the runner was declared lost, taking the upload and any account of the failure with it. Head and tail only, CP filtered out, wrapped in timeout, and the artifact names its files rather than sweeping up whatever the directory holds. KVM needed no change: the action already fails the job outright when /dev/kvm is absent, which is correct here because qemu-tool has no TCG fallback to degrade into. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
The emulated device backs ~43 MB of scratch, which works out to roughly 1300 B/thread of private segment before a dispatch stops fitting. Each nvme-ep per-batch array entry costs 40 B/thread, so the guest script's BATCH_DEPTH=32 pinned 1280 B -- effectively the entire budget, with nothing left for the kernel's own locals and spills. Single-queue tests squeaked under that. Multi-queue ones did not, because ROCr provisions scratch for full occupancy per queue rather than by grid size, so run 35380016526 failed nvme-smoke-queues-4 with HSA_STATUS_ERROR_OUT_OF_RESOURCES gpuKernel sync failed for queue 5: too many resources requested for launch (code: 701) and nvme-ep-num-queues-4 with "Failed to create HIP stream for queue 8: out of memory". Those are the failures vfio-rocjitsu's handoff report predicts for this image and calls rocm-xio's to fix, not the image's: patch 0005 makes the CP ask the guest runtime for scratch at all, but the ceiling itself is a property of the emulation and no patch there moves it. CI is already on the newest published rocjitsu tag, so there was nothing to bump. Halve the default to 16, or 640 B/thread. This bounds submission batch depth, not I/O count, so nvme-smoke-batch-128 still issues its 128 I/Os, in more and smaller batches. Echo both emulation knobs at configure time: the previous run's log recorded neither, so which depth produced which result had to be inferred from the script's default. Drop the stress label on pull requests while here. It is three tests and 994.6 s of a 3845 s ctest, and nvme-qid-recreate-gpu-stress alone is 866 s -- 23% of the suite for one test. They are also the slowest per unit of information: the qid tests spend ~40 s an iteration on controller resets and 1536-read warm-ups wrapped around an operation that resolves in 0.11 s. The weekly schedule and manual runs still get them, and CTEST_EXCLUDE_LABEL takes any regex for a narrower cut. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
Collaborator
Author
|
@copilot update this based on the latest images and also review the QEMU-minimal workflows that do similar things. |
Contributor
|
@sbates130272 I'm unable to start working on this because of repository rules that prevent me from pushing to the branch:
See the documentation for more details. |
Collaborator
Author
|
Yes do it as a PR please. |
Collaborator
Author
|
Superceeded by #220. Closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
nvme-ep has had no hardware-free CI path. build-check compiles it and test-emulate exercises test-ep in emulate mode, but nothing drives real NVMe queues from device code, so the submission and completion paths only ever ran on the test node.
This adds a workflow that boots a QEMU guest with a rocjitsu emulated gfx1250 attached over vfio-user and an emulated NVMe controller behind the pci-mmio-bridge, then builds rocm-xio inside it and runs the nvme-ep ctest labels against that controller.
The pieces:
.github/actions/rocjitsu-nvme-vm/ composite action that boots the VM
.github/compose/rocjitsu-nvme-vm/ the rocjitsu + QEMU compose stack
scripts/test/ansible/ guest ROCm/amdgpu provisioning
scripts/test/vm-provision.sh runner-side playbook driver
scripts/test/vm-guest-test.sh in-guest build and test driver
.github/workflows/test-vm-nvme.yml the workflow tying them together
Bring-up is a composite action rather than inline steps because the same VM is wanted by rocm-ernic and anything else without hardware to test against. It is kept free of rocm-xio specifics so it can move to sbates130272/qemu-minimal, next to qemu-tool and the other compose stacks, once it has proven itself here; consumers then switch to a
uses:reference and this directory goes away.The QEMU image must be the -sbates-fork build: pci-mmio-bridge lives on dev/stephen/pci-mmio-bridge-submit and is not upstream, and qemu-tool hard-fails if the binary does not advertise the device. That image's entrypoint.sh ignores "$@" and execs a fixed qemu-system line, so the stack overrides the entrypoint to go through qemu-tool, which is what supplies --nvme and --pci-mmio-bridge.
Every image is pinned to a dated tag, collected in one env: block so a bump is a one-line reviewable change. :latest is deliberately not used: the rocjitsu pin tracks a force-pushed review branch, and a silent retag would turn a run red with no diff to point at.
Guest provisioning does not reuse qemu-minimal's vm-rocjitsu.yml. That playbook still calls vfio_guest_firmware.py, which the gfx1250-vfio- compute stack removed in favour of real firmware from the driver release matching the guest's amdgpu.ko. The playbook here asks the built module what it wants via modinfo -F firmware and fails with a list of missing blobs, rather than leaving the guest to hang in gfx_v12_1_hw_init.
Whether therock plus amdgpu-dkms actually ships that gfx1250 firmware inventory on resolute is untested and can only be settled by a real run. The workflow is therefore workflow_dispatch plus a weekly schedule, not a pull_request trigger: a full ROCm and DKMS provision under emulation takes the better part of an hour, and the rocjitsu pin is not yet stable. workflow_dispatch accepts skip-gpu=1 for a build-only smoke run while that settles.