fix(vm): unpack registry images correctly and validate prepared disks - #3524
Conversation
The registry image-prep path expected `umoci raw unpack` to produce a bundle-style rootfs/ subdirectory, but it extracts the image filesystem directly into the target. Every registry prep therefore failed after a successful unpack. Guest init exit codes do not survive the libkrun boundary, so the failure looked like success and the broken disk was cached, making every later sandbox for that image fail with "prepared image disk missing /image-rootfs". VM E2E started hitting this after the bootstrap image moved to nvcr.io/nvidia/base/ubuntu:24.04: `--from base` no longer matches the bootstrap image, so it now goes through registry prep. - Accept umoci's direct extraction layout in the guest prep script. - Build the image rootfs under a partial directory and rename it to /image-rootfs only after every prep step succeeds. - Check the prepared disk for /image-rootfs before caching it. On failure, leave the cache untouched and report the prep console tail. - Size the prep disk to hold the payload and the unpacked rootfs at the same time. The community base image needs 1.40 GB + 3.32 GB, which did not fit in the old payload*3 + 512 MiB. Fixes NVIDIA#2358 Co-authored-by: s2cube <26961336+s2cube@users.noreply.github.com> Signed-off-by: Emilien Macchi <emacchi@redhat.com>
|
/ok to test af3c7f3 |
|
Label |
The host_gateway_alias and vm_corporate_proxy workloads run curl and python3. The VM driver now defaults to nvcr.io/nvidia/base/ubuntu:24.04, which ships neither, so these tests fail in VM E2E with "command not found". Request the community base image explicitly with `--from base`. Docker, Podman, and Kubernetes E2E already default to that image, so their behavior is unchanged. Signed-off-by: Emilien Macchi <emacchi@redhat.com>
|
/ok to test 3cd2ddf |
PR Review StatusThis focused fix for the confirmed VM image-preparation bug in #2358 is project-valid. The independent initial review found no blocking defects or non-blocking suggestions in the current patch. Action required: A maintainer should inspect or re-run the failed Blocking findings:
Carried findings:
Non-blocking suggestions:
Gator metadata
|
Monitoring CompleteMonitoring is complete because this PR has merged. Final status: The initial code review found no blocking findings. The VM E2E jobs passed; the aggregate E2E gate was still reporting a separate Docker Python job failure at the last pre-merge check. I removed the active |
3 participants
Summary
Fix VM registry image prep, which has failed for every image other than the bootstrap image since
umoci raw unpackwas introduced. The failure was masked as success and the broken disk was cached. VM E2E started hitting it after #3446: the bootstrap image is nownvcr.io/nvidia/base/ubuntu:24.04, so--from basegoes through registry prep and every sandbox fails withFATAL: prepared image disk missing /image-rootfs.A second commit gets VM E2E green again: the tests behind conformance needed
curl/python3, which the new default image doesn't ship.Related Issue
Fixes #2358
Builds on #2360 by @s2cube, which the vouch gate auto-closed. Its
driver.rschanges no longer applied cleanly, so this is a rework with co-author credit.Changes
fix(vm): unpack registry images correctly and validate prepared disksumoci raw unpackextracts directly into the target (no bundle-stylerootfs/); verified with umoci v0.6.0 on the community base image. The rootfs is now built underimage-rootfs.partialand renamed to/image-rootfsonly after every step succeeds (unpack or tar extract, runtime install, identity marker, payload cleanup)./image-rootfswith a read-onlydebugfs stat. On failure nothing is cached and the error carries the prep console tail.payload*3 + 512 MiBtopayload*4 + 1 GiB. The community base image needs 1.40 GB (OCI payload) + 3.32 GB (unpacked blocks), which didn't fit in the old 4.74 GB before ext4 metadata. The disk file is sparse, so the headroom costs no host disk.test(e2e): run tool-dependent VM tests from the community base imagehost_gateway_alias(curl) andvm_corporate_proxy(python3) now create their sandboxes with--from base. With bug(vm-driver): registry image prep expects bundle-style rootfs/ fromumoci raw unpack— failure masked as success and broken disk cached forever #2358 fixed, CI got past conformance and failed there withcurl: command not found;vm_corporate_proxywould have failed next onpython3.--from baseresolves toghcr.io/nvidia/openshell-community/sandboxes/base:latest, and Podman's digest-pinned mode tags its pinned image under that name), so their behavior is unchanged.Testing
mise run pre-commitpassesopenshell-driver-vm: 171 passed, clippy clean)Local
mise run e2e:vmwith the new default image (KVM, image not in local podman, so the registry path runs like CI) passes end to end:smoke,sandbox-lifecycle),host_gateway_alias(2/2),vm_overlay(1/1),vm_gateway_start(1/1),vm_corporate_proxy(3/3, including the fake-proxy routes).sha256:aeef1c63…), contains a complete/image-rootfswith the identity marker, and has no leftover.partialdirectory.Reviews: two adversarial review agents (correctness and architecture) ran on the fix commit and their confirmed findings are applied (rename-last in the guest script, non-ext4 rejection test). The fix commit was also reviewed by gpt-6-astra (xhigh). The test commit is a small argument change, validated by the E2E run above.
Checklist
🤖 Generated with Claude Code