|
| 1 | +<!-- |
| 2 | +SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. |
| 3 | +SPDX-License-Identifier: Apache-2.0 |
| 4 | +--> |
| 5 | + |
| 6 | +# Test Guests |
| 7 | + |
| 8 | +This prototype uses Nix, QEMU, and Ansible to boot and configure disposable Linux VMs for testing OpenShell packages and binaries. It supports HVF on Apple Silicon macOS, KVM on native-architecture Linux hosts, and a slower TCG fallback on Linux when KVM is unavailable. |
| 9 | + |
| 10 | +## Requirements |
| 11 | + |
| 12 | +- Nix with flakes enabled. |
| 13 | +- Apple Silicon macOS with HVF, or a native-architecture Linux host. Linux uses KVM when `/dev/kvm` is available and falls back to QEMU TCG otherwise. |
| 14 | +- Enough local capacity for a four-vCPU, 4 GiB guest and a disposable disk overlay. |
| 15 | +- Native-architecture artifacts. TCG emulates the guest CPU on Linux but does not enable cross-architecture guests. |
| 16 | + |
| 17 | +The first run downloads the selected cloud image and VM runtime. Nix reuses those immutable inputs on later runs, while each guest starts from a fresh writable overlay. |
| 18 | + |
| 19 | +## Directory structure |
| 20 | + |
| 21 | +```text |
| 22 | +nix/test-guest/ |
| 23 | +├── README.md |
| 24 | +├── default.nix |
| 25 | +├── run.sh |
| 26 | +├── cache.sh |
| 27 | +├── cache-lib.sh |
| 28 | +├── cache-seal.sh |
| 29 | +├── distros/ |
| 30 | +│ ├── ubuntu.nix |
| 31 | +│ ├── centos.nix |
| 32 | +│ ├── fedora.nix |
| 33 | +│ └── rocky.nix |
| 34 | +└── configuration/ |
| 35 | + ├── docker.yml |
| 36 | + ├── podman.yml |
| 37 | + └── selinux.yml |
| 38 | +``` |
| 39 | + |
| 40 | +- `default.nix` assembles the guest and cache flake apps. It selects host architecture and acceleration, supplies the runtime tools, and exposes distro profiles and configuration playbooks as Nix-store catalogs. |
| 41 | +- `run.sh` owns the disposable guest lifecycle: cache lookup, cloud-image realization, cloud-init seed creation, QEMU startup, SSH readiness, Ansible execution, artifact installation, guest command execution, and cleanup. |
| 42 | +- `cache.sh` ensures an exact prepared disk exists locally. It can pull or explicitly push the disk as an OCI artifact. |
| 43 | +- `cache-lib.sh` defines deterministic cache identity and validation helpers shared by the runner and cache command. |
| 44 | +- `cache-seal.sh` removes per-instance state and zeroes free space inside a prepared guest before capture. |
| 45 | +- `distros/*.nix` define the immutable base-image catalog. Each record pins and exports the image URL and hash and declares the expected OS ID, version, and package family. |
| 46 | +- `configuration/*.yml` are host-executed Ansible playbooks that layer optional capabilities onto a base guest. Configurations remain independent and run in the order supplied with repeated `--with` arguments. |
| 47 | +- `README.md` documents the supported combinations and developer interface. |
| 48 | + |
| 49 | +The root [`flake.nix`](../../flake.nix) exposes this directory as the `test-guest` and `test-guest-cache` apps. Debian artifact creation remains outside the guest harness in [`tasks/scripts/package-deb.sh`](../../tasks/scripts/package-deb.sh); the runner only installs or copies artifacts that already exist. |
| 50 | + |
| 51 | +## Supported configurations |
| 52 | + |
| 53 | +| Distro | Docker | Podman | SELinux | Package format | |
| 54 | +| --- | --- | --- | --- | --- | |
| 55 | +| Ubuntu 24.04 | Yes | Yes | No | `.deb` | |
| 56 | +| CentOS Stream 10 | No | Yes | Yes | `.rpm` | |
| 57 | +| Fedora 44 | No | Yes | Yes | `.rpm` | |
| 58 | +| Rocky Linux 9 | Yes | Yes | Yes | `.rpm` | |
| 59 | + |
| 60 | +List the available distros and configurations: |
| 61 | + |
| 62 | +```shell |
| 63 | +nix run .#test-guest -- --list |
| 64 | +``` |
| 65 | + |
| 66 | +## Open an interactive VM |
| 67 | + |
| 68 | +Boot a base Ubuntu VM: |
| 69 | + |
| 70 | +```shell |
| 71 | +nix run .#test-guest -- --distro ubuntu |
| 72 | +``` |
| 73 | + |
| 74 | +Apply the Docker configuration before opening the SSH session: |
| 75 | + |
| 76 | +```shell |
| 77 | +nix run .#test-guest -- --distro ubuntu --with docker |
| 78 | +``` |
| 79 | + |
| 80 | +Other combinations use the same interface: |
| 81 | + |
| 82 | +```shell |
| 83 | +nix run .#test-guest -- --distro rocky --with docker |
| 84 | +nix run .#test-guest -- --distro centos --with podman |
| 85 | +nix run .#test-guest -- --distro fedora --with podman |
| 86 | +``` |
| 87 | + |
| 88 | +Configurations are repeatable: |
| 89 | + |
| 90 | +```shell |
| 91 | +nix run .#test-guest -- \ |
| 92 | + --distro ubuntu \ |
| 93 | + --with docker \ |
| 94 | + --with podman |
| 95 | +``` |
| 96 | + |
| 97 | +Ensure SELinux is enforcing on CentOS, Fedora, or Rocky: |
| 98 | + |
| 99 | +```shell |
| 100 | +nix run .#test-guest -- \ |
| 101 | + --distro rocky \ |
| 102 | + --with docker \ |
| 103 | + --with selinux \ |
| 104 | + -- getenforce |
| 105 | +``` |
| 106 | + |
| 107 | +`--with selinux` installs the required tooling, persists `SELINUX=enforcing`, applies enforcing mode live, and verifies the result. It fails on Ubuntu and on guests where SELinux is fully disabled and would require a reboot to enable. |
| 108 | + |
| 109 | +## Ansible configurations |
| 110 | + |
| 111 | +Configurations are Ansible playbooks stored under `nix/test-guest/configuration/`. Ansible runs on the host using the VM's ephemeral SSH key and loopback port. The guest does not install Ansible. |
| 112 | + |
| 113 | +Configurations run in the order provided on the command line. OpenShell packages and copied binaries are installed after all configurations succeed. |
| 114 | + |
| 115 | +`--install` packages and `--copy` executables are applied by a dedicated per-run Ansible playbook. They are not stored in prepared VM cache entries. |
| 116 | + |
| 117 | +## Prepared VM cache |
| 118 | + |
| 119 | +The `test-guest-cache` app ensures a prepared disk exists for one exact distro, host architecture, and ordered configuration list. It checks the local cache first, optionally pulls a matching OCI artifact, or builds and validates a new local entry on a miss: |
| 120 | + |
| 121 | +```shell |
| 122 | +nix run .#test-guest-cache -- \ |
| 123 | + --distro ubuntu \ |
| 124 | + --with docker |
| 125 | +``` |
| 126 | + |
| 127 | +Configure an OCI repository and a trusted manifest digest to use it as a shared |
| 128 | +backing cache: |
| 129 | + |
| 130 | +```shell |
| 131 | +nix run .#test-guest-cache -- \ |
| 132 | + --distro ubuntu \ |
| 133 | + --with docker \ |
| 134 | + --repository ghcr.io/nvidia/openshell/test-guest-cache \ |
| 135 | + --digest sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef |
| 136 | +``` |
| 137 | + |
| 138 | +The command never publishes implicitly. Add `--push` after authenticating ORAS through its Docker-compatible credential configuration: |
| 139 | + |
| 140 | +```shell |
| 141 | +nix run .#test-guest-cache -- \ |
| 142 | + --distro ubuntu \ |
| 143 | + --with docker \ |
| 144 | + --repository ghcr.io/nvidia/openshell/test-guest-cache \ |
| 145 | + --push |
| 146 | +``` |
| 147 | + |
| 148 | +A successful push prints the immutable `repository@sha256:...` reference. Supply |
| 149 | +that digest to consumers through trusted CI configuration. Pulls by mutable tag |
| 150 | +are not allowed. A pulled local entry records its manifest digest and is reused |
| 151 | +only when it matches the requested trusted digest. |
| 152 | + |
| 153 | +A cache build boots and configures a disposable VM, runs the internal sealing script, flattens the overlay into a standalone QCOW2 disk, and validates a fresh boot before committing the entry. The OCI artifact contains metadata and a `disk.qcow2.zst` layer. |
| 154 | + |
| 155 | +The key includes the pinned base-image identity, guest architecture, ordered configuration file digests, Ansible version, cache generation, and sealing script digest. Installed packages, copied binaries, forwarded ports, and guest commands are never cached. |
| 156 | + |
| 157 | +Normal `test-guest` runs automatically use an exact valid local entry after |
| 158 | +rechecking its disk checksum and QCOW2 structure. On a local miss, the runner |
| 159 | +invokes the cache builder and stores the prepared disk before continuing. It |
| 160 | +then creates a fresh writable overlay, cloud-init instance, machine ID, and SSH |
| 161 | +identity from that entry. Set `OPENSHELL_TEST_GUEST_CACHE_DISABLE=1` to bypass |
| 162 | +both local lookup and automatic population. |
| 163 | + |
| 164 | +The default cache directory is `${XDG_CACHE_HOME:-$HOME/.cache}/openshell/test-guest`. Override it with `--cache-dir` on the cache command or `OPENSHELL_TEST_GUEST_CACHE_DIR` for either app. |
| 165 | + |
| 166 | +Cache command options: |
| 167 | + |
| 168 | +```text |
| 169 | +--distro NAME Base distro: ubuntu, centos, fedora, or rocky |
| 170 | +--with NAME Apply docker, podman, or selinux; repeatable |
| 171 | +--repository REF OCI repository without a tag |
| 172 | +--digest DIGEST Trusted OCI manifest digest required for pulls |
| 173 | +--cache-dir PATH Override the local prepared-disk cache directory |
| 174 | +--push Publish the ensured entry to the repository |
| 175 | +``` |
| 176 | + |
| 177 | +## Install an OpenShell package |
| 178 | + |
| 179 | +Package existing ARM64 Linux binaries with the repository's `package:deb:arm64` mise task: |
| 180 | + |
| 181 | +```shell |
| 182 | +OPENSHELL_CLI_BINARY="$PWD/target/aarch64-unknown-linux-musl/release/openshell" \ |
| 183 | +OPENSHELL_GATEWAY_BINARY="$PWD/target/aarch64-unknown-linux-gnu/release/openshell-gateway" \ |
| 184 | +OPENSHELL_DRIVER_VM_BINARY="$PWD/target/aarch64-unknown-linux-gnu/release/openshell-driver-vm" \ |
| 185 | +OPENSHELL_DEB_VERSION=0.0.0-local \ |
| 186 | +OPENSHELL_OUTPUT_DIR="$PWD/artifacts" \ |
| 187 | +nix develop --command mise run package:deb:arm64 |
| 188 | +``` |
| 189 | + |
| 190 | +Install the package in an Ubuntu VM and run a command: |
| 191 | + |
| 192 | +```shell |
| 193 | +nix run .#test-guest -- \ |
| 194 | + --distro ubuntu \ |
| 195 | + --with docker \ |
| 196 | + --install artifacts/openshell_0.0.0-local_arm64.deb \ |
| 197 | + -- openshell --version |
| 198 | +``` |
| 199 | + |
| 200 | +For an x86_64 Linux guest, supply x86_64 binaries and use `package:deb:amd64`. The package architecture must match the host and guest architecture. |
| 201 | + |
| 202 | +`--install` is repeatable. Debian packages are accepted by Ubuntu; RPM packages are accepted by CentOS, Fedora, and Rocky Linux. This prototype can install an existing RPM but does not build one. |
| 203 | + |
| 204 | +## Copy binaries directly |
| 205 | + |
| 206 | +Use `--copy SOURCE:DEST` to install an executable without creating a package: |
| 207 | + |
| 208 | +```shell |
| 209 | +nix run .#test-guest -- \ |
| 210 | + --distro ubuntu \ |
| 211 | + --copy ./openshell:/usr/local/bin/openshell \ |
| 212 | + -- openshell --version |
| 213 | +``` |
| 214 | + |
| 215 | +The destination must be an absolute guest path. Copied files are installed with mode `0755`. |
| 216 | + |
| 217 | +## Runner options |
| 218 | + |
| 219 | +```text |
| 220 | +--distro NAME Base distro: ubuntu, centos, fedora, or rocky |
| 221 | +--with NAME Apply docker, podman, or selinux; repeatable |
| 222 | +--install PATH Install a .deb or .rpm package; repeatable |
| 223 | +--copy SRC:DEST Copy an executable into the guest; repeatable |
| 224 | +--ssh-port PORT Use a specific loopback SSH forwarding port |
| 225 | +--forward-port HOST_PORT:GUEST_PORT |
| 226 | + Forward a loopback host port to a guest port; repeatable |
| 227 | +--keep Preserve the disk overlay and logs after shutdown |
| 228 | +--list List distros and configurations |
| 229 | +``` |
| 230 | + |
| 231 | +Each `--forward-port` binds only `127.0.0.1` on the host. Both ports must be unprivileged values from 1024 through 65535, and each host port may appear only once. |
| 232 | + |
| 233 | +Arguments after `--` are executed inside the guest. Without a command, the runner opens an interactive SSH session. |
| 234 | + |
| 235 | +## Lifecycle |
| 236 | + |
| 237 | +Each invocation ensures an exact prepared local cache entry exists. On a miss, |
| 238 | +the cache builder realizes the hash-pinned cloud image, applies the selected |
| 239 | +configurations, seals and validates the prepared disk, and stores it locally. |
| 240 | +The runner then: |
| 241 | + |
| 242 | +1. Creates a temporary QCOW2 overlay backed by the prepared cache disk or pinned cloud image. |
| 243 | +2. Boots QEMU with HVF, KVM, or the Linux TCG fallback. |
| 244 | +3. Creates a fresh cloud-init instance and ephemeral SSH key. |
| 245 | +4. Applies the selected Ansible configurations only when the base is not prepared. |
| 246 | +5. Installs or copies the supplied artifacts. |
| 247 | +6. Opens SSH or executes the requested guest command. |
| 248 | +7. Powers off QEMU and deletes the writable overlay. |
| 249 | + |
| 250 | +Prepared cache disks remain read-only. Test-specific state exists only in the disposable overlay. |
| 251 | + |
| 252 | +Use `--keep` to preserve the overlay, cloud-init seed, SSH key, and serial log for debugging. The retained directory is printed when the runner exits. |
| 253 | + |
| 254 | +## Current limitations |
| 255 | + |
| 256 | +- Host and guest architectures must match. |
| 257 | +- TCG is slower than hardware virtualization and uses a longer SSH readiness timeout. |
| 258 | +- Prepared cache entries are architecture-specific and match the exact ordered configuration list. |
| 259 | +- OCI pulls transfer a complete compressed standalone disk; incremental disk layers are not implemented. |
| 260 | +- Guest ports are reachable from the host only when explicitly exposed with loopback-only `--forward-port`. |
| 261 | +- The runner does not build OpenShell, configure a gateway, or select an E2E test suite. |
0 commit comments