Skip to content

qemu.md

Latest commit

 

History

History
330 lines (266 loc) · 19.4 KB

qemu.md

File metadata and controls

330 lines (266 loc) · 19.4 KB
Raw

qemu-coreboot-(fb)whiptail-tpmX(-hotp) boards

The qemu-coreboot-fbwhiptail-tpm1-hotp configuration (and their variants) permits testing of most features of Heads.
It requires a supported USB token (which will be reset for use with the VM, do not use a token needed for a real machine). With KVM acceleration, speed is comparable to a real machine. If KVM is unavailable, lightweight desktops are still usable.

Heads is currently unable to reflash firmware within qemu, which means that OEM reset and re-ownership cannot be fully performed within the VM. Instead, a GPG key can be injected in the Heads image from the host during the build.

The TPM state, virtual disks, and token state for this configuration are persisted under build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/ by default.

Bootstrapping a working system

Important: The supported and tested workflow uses the provided Docker wrappers (./docker_repro.sh, ./docker_local_dev.sh, or ./docker_latest.sh). Host-side installation of QEMU, swtpm, or other QEMU-related tooling is unnecessary and is not part of the standard, supported workflow; only advanced or edge-case scenarios should install those tools on the host (see 'Troubleshooting' below for guidance).

  1. Install Docker

Note: the Nix-built Docker images used by ./docker_repro.sh include QEMU (qemu-system-x86_64), swtpm / libtpms, canokey-qemu (a virtual OpenPGP smartcard), and other userspace tooling required to build and test QEMU targets. These images are intended to be self-contained for QEMU testing; host-focused build instructions (e.g., building swtpm on the host) were removed to avoid divergence—use the Docker wrappers for the tested workflow.

If you do not specify USB_TOKEN when running QEMU targets, the container will use the included canokey-qemu virtual token by default. To forward a hardware token from the host, set USB_TOKEN or pass hostbus/hostport/vendorid,productid to the make invocation.

If you plan to manage disk images on the host (outside the container), install qemu-utils for qemu-img. Mounting uses losetup (from util-linux, present on all Linux distributions):

sudo losetup --find --show --partscan \
  ./build/x86/<board>/usb_fd.raw
# → /dev/loopN; use sudo fdisk -l /dev/loopN to check partitions
sudo mount /dev/loopNp1 /mnt   # partitioned, or /dev/loopN if flat

The Makefile creates a sparse usb_fd.raw with an MBR partition table and ext4 filesystem. Its 64G default is virtual capacity, not its initial host allocation; host usage grows as data is written. Older images may be flat — check with sudo fdisk -l first.

Note: the Docker container bind-mounts the current worktree at the same path and passes selected X11, KVM, and USB devices separately. Arbitrary host paths are not thereby visible, so build data, keys, disk images, and installer images must resolve inside the worktree. Use qemu_img/ inside the repository as a backing store (see the hardlink workflow below).

  1. Build Heads

    • ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp

  2. Optionally attach an installer ISO (this attaches the ISO in place of the normal virtual USB flash; it does not run the installer automatically)

    • ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp INSTALL_IMG=./qemu_img/installer.iso run
    • Lightweight desktops (XFCE, LXDE, etc.) are recommended, especially if KVM acceleration is not available (such nested in Qubes OS)
    • When running nested in a qube, disable memory ballooning for the qube, or performance will be very poor.
    • QEMU_MEMORY_SIZE is recorded in the board's memory file only when that file is first created; delete it to change the guest-memory setting. The default is 4G.
    • QEMU_DISK_SIZE applies when root.qcow2 is first created; delete/recreate that image to change its virtual size. The default is 20G.
    • QEMU_USB_SIZE likewise applies when usb_fd.raw is first created; delete/recreate it to change the USB-image size.

  3. Shut down and boot Heads with the USB token attached, proceed with OEM reset

    • ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp USB_TOKEN=<token> run
    • If you do not set USB_TOKEN, the included canokey-qemu virtual token will be used by default.
    • For <token>, use one of:
      • NitrokeyPro - a Nitrokey Pro by VID/PID
      • NitrokeyStorage - a Nitrokey Storage by VID/PID
      • Nitrokey3NFC - a Nitrokey 3 by VID:PID
      • LibremKey - a Librem Key by VID/PID
      • hostbus=#,hostport=# - indicate a host bus and port (see qemu usb-host)
      • vendorid=#,productid=# - indicate a device by VID/PID (decimal, see qemu usb-host)
    • You do need to export the GPG key to a USB disk, otherwise defaults are fine.
    • Head will show an error saying it can't flash the firmware, continue
    • Then Heads will indicate that there is no TOTP code yet, at this point shut down (Continue to main menu -> Power off)

  4. Get the public key that was saved to the virtual USB flash drive

    • sudo mkdir /media/fd_heads_gpg

    • Attach the raw image and mount the public partition:

      sudo losetup --find --show --partscan
      ./build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/usb_fd.raw

      → prints /dev/loopN with /dev/loopNp1, /dev/loopNp2, etc.

      sudo fdisk -l /dev/loopN # verify partition layout sudo mount /dev/loopNp2 /media/fd_heads_gpg # second/public partition

    • mkdir -p qemu_img

    • Copy the most recent public key from /media/fd_heads_gpg into the worktree:

    • cp /media/fd_heads_gpg/<key-file> ./qemu_img/public-key.asc

    • sudo umount /media/fd_heads_gpg

    • sudo losetup -d /dev/loopN # detach (replace N with the actual number)

  5. Inject the GPG key into the Heads image and run again

    • ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp PUBKEY_ASC=./qemu_img/public-key.asc inject_gpg
    • ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp USB_TOKEN=LibremKey PUBKEY_ASC=./qemu_img/public-key.asc run

  6. Initialize the TPM - select "Reset the TPM" at the TOTP error prompt and follow prompts

  7. Select "Default boot" and follow prompts to sign /boot for the first time and set a default boot option

You can reuse an existing root disk by passing ROOT_DISK_IMG and a path inside the current worktree. The default is a QCOW2 image named root.qcow2:

./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp \
  PUBKEY_ASC=./qemu_img/public-key.asc \
  USB_TOKEN=NitrokeyStorage \
  ROOT_DISK_IMG=./qemu_img/root.qcow2 \
  run

Saving Disk Images from Build-Dir Wipes

The current worktree is bind-mounted at the same path inside the container (docker/common.sh: -v "$(pwd):$(pwd)"); X11 sockets, KVM, and USB devices are passed separately. Build-data paths must still resolve inside that worktree. Any backup copy used by the build must live inside it — ~/QemuImages/ and other user-home paths are not exposed by the worktree bind mount.

QEMU state under build/x86/<board>/ persists only in the same worktree when ordinary rebuilds/checkouts do not run a repo-wide git clean. A new clone has no copied build state. build/ is ignored, so a normal checkout does not remove an existing state tree, but git clean -fxd can remove untracked qemu_img/ data unless it is protected or backed up. The disk/token files are also removed by named cleanup helpers that remove the relevant build tree, or by manual deletion. Use hardlinks to keep backups inside the worktree and share them across board variants:

mkdir -p qemu_img                         # safe storage inside clone
cp build/x86/<board>/root.qcow2 qemu_img/ # copy OS install to safety
rm build/x86/<board>/root.qcow2           # remove build-tree copy
cp -alf qemu_img/root.qcow2 build/x86/<board>/  # hardlink back
# Now both paths point to the same data on disk.
# Wiping build/ won't touch qemu_img/.

# Restore after a wipe:
cp -alf qemu_img/root.qcow2 build/x86/<board>/root.qcow2
cp -alf qemu_img/usb_fd.raw  build/x86/<board>/usb_fd.raw

cp -alf creates a hardlink — a second directory entry pointing to the same data blocks (zero additional space). Data is freed only when the last link is deleted. Caveat: writes to one link affect all links. Use qemu-img snapshot before modifying the root disk.

USB flash drive workflow

mkdir -p qemu_img                             # safe storage inside clone

# Step 1: Create the USB image via the Makefile, then launch QEMU.
./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 \
  QEMU_USB_SIZE=64G run
# → build/x86/.../usb_fd.raw now exists; the command also starts QEMU.

# Step 2: Save a master copy IMMEDIATELY (before population).
cp build/x86/qemu-coreboot-fbwhiptail-tpm2/usb_fd.raw qemu_img/usb_fd.raw
rm build/x86/qemu-coreboot-fbwhiptail-tpm2/usb_fd.raw
cp -alf qemu_img/usb_fd.raw build/x86/qemu-coreboot-fbwhiptail-tpm2/usb_fd.raw

# Step 3: Populate with ISOs stored inside the clone.
mkdir -p qemu_img/isos
sudo losetup --find --show --partscan build/x86/.../usb_fd.raw
sudo mount /dev/loop0p1 /mnt
cp qemu_img/isos/*.iso /mnt/
sudo umount /mnt && sudo losetup -d /dev/loop0

# Step 4: Hardlink into other board build directories.
cp -alf qemu_img/usb_fd.raw build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/usb_fd.raw
cp -alf qemu_img/usb_fd.raw build/x86/qemu-coreboot-fbwhiptail-tpm2-hotp/usb_fd.raw

# Next run uses the hardlink — Makefile skips creation since the file exists.

Daily development cycle

After OS install + USB provisioned, reference both from ./qemu_img/:

./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp \
  PUBKEY_ASC=./qemu_img/public-key.asc \
  USB_TOKEN=Nitrokey3NFC \
  ROOT_DISK_IMG=./qemu_img/root.qcow2 \
  inject_gpg run

Running via Docker wrappers

We provide convenient wrapper scripts at the repository root that encapsulate Docker invocation and automatically handle common host integrations needed for QEMU runs.

Wrapper comparison

Script Image Use
docker_latest.sh Defaults to pinned digest when available Convenience: run the latest published image
docker_local_dev.sh linuxboot/heads:dev-env Development: use local image built from the flake (rebuilds when flake files are dirty)
docker_repro.sh DOCKER_REPRO_DIGEST environment value, falling back to docker/DOCKER_REPRO_DIGEST On the default canonical repository path, the resolved digest is cross-checked against .circleci/config.yml; fork/noncanonical overrides can skip that check. The wrapper does not compare ROMs or hashes.txt with CI

What the wrappers handle

Wrapper options: some runtime behavior is controlled via environment variables documented in the repository README (see 'Wrapper options & environment variables'). Wrapper scripts now have focused --help output for their own variables, and ./docker/common.sh prints the full environment reference. Important ones are HEADS_DISABLE_USB (set to 1 to disable automatic USB passthrough and cleanup) and HEADS_X11_XAUTH (force mounting your $HOME/.Xauthority).

Make variables such as USB_TOKEN, PUBKEY_ASC, INSTALL_IMG, QEMU_MEMORY_SIZE, QEMU_DISK_SIZE, QEMU_USB_SIZE, ROOT_DISK_IMG, CPUS and V are forwarded to the make invocation and affect how targets/qemu.mk runs QEMU. QEMU_MEMORY_SIZE is written only when the board's memory file is first created; delete that file to apply a new value. See targets/qemu.mk for token formats and examples.

The two default virtual disks have different formats:

  • root.qcow2 is the QCOW2 root disk. It is created with qemu-img create -f qcow2; its default virtual size is 20G. QEMU_DISK_SIZE is used only when this file is first created; delete/recreate the image to change the size.
  • usb_fd.raw is a raw virtual USB flash image. It is created with qemu-img create -f raw, partitioned, and formatted; its default virtual size is 64G. QEMU_USB_SIZE is used only on first creation; delete/recreate the image to change its size. It is not converted to QCOW2.

Both are created under build/x86/<BOARD>/. Both formats can be sparse: actual host allocation grows only as blocks are written, so virtual capacity is not the initial host-disk footprint. For example, INSTALL_IMG=./qemu_img/installer.iso substitutes an image inside the worktree for usb_fd.raw for the QEMU run.

See Saving Disk Images from Build-Dir Wipes above for the recommended workflow: save a master copy immediately, hardlink it back into the build directory, then populate it (which modifies the shared blocks).

Quick mount reference (after following the hardlink workflow):

mkdir -p qemu_img/isos
sudo losetup --find --show --partscan \
  build/x86/qemu-coreboot-fbwhiptail-tpm2/usb_fd.raw
# → /dev/loopN; check partitions with sudo fdisk -l /dev/loopN
sudo mount /dev/loopNp1 /mnt   # or /dev/loopN if flat
cp qemu_img/isos/*.iso /mnt/
sudo umount /mnt && sudo losetup -d /dev/loopN

Note: when USB passthrough is active the wrapper will warn and, on
interactive shells, give a 3s abort window before attempting to kill
processes that hold the token (e.g., `scdaemon`/`pcscd`) to free the
device; set `HEADS_DISABLE_USB=1` to opt out.

- **KVM passthrough**: when `/dev/kvm` exists on the host the container is run with `/dev/kvm` mounted into the container, enabling KVM-accelerated QEMU.
- **X11 GUI support**: the wrappers mount the X11 socket and programmatically create a temporary Xauthority file (via `mktemp -t heads-docker-xauth-XXXXXX`, or `/tmp/.docker.xauth-<uid>` as fallback when mktemp is unavailable) when `xauth` is available; they fall back to mounting `${HOME}/.Xauthority` when needed and set `XAUTHORITY` inside the container so GTK/SDL QEMU windows work. The temp file is cleaned up automatically after `docker run` completes.
  - To force mounting your `${HOME}/.Xauthority` regardless of socket detection, set `HEADS_X11_XAUTH=1`.
- **USB passthrough**: when host USB buses exist `/dev/bus/usb` is mounted into the container so VMs can access hardware tokens. To explicitly disable automatic USB passthrough set `HEADS_DISABLE_USB=1`.
- **USB token cleanup**: the wrappers attempt to detect and stop local GPG/toolstack processes (e.g., `scdaemon`, `pcscd`) which might hold USB tokens. Behavior notes:
  - If `sudo` can be run without a password the cleanup runs silently.
  - The cleanup avoids prompting for a password in non-interactive shells; it will prompt only when running interactively (attached to a TTY). To skip the cleanup entirely set `HEADS_DISABLE_USB=1`.
- **Convenience variables accepted by the wrappers**: `V=1` for verbose make output, `CPUS=N` to set parallelism for builds, and any `make` variables may be passed through to the container command.
- **Argument forwarding**: arguments given to the wrapper are forwarded directly to the container command (no special separator needed). For example: `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run`.

Environment variables reference
---

| Variable | Default | Effect |
|---|---:|---|
| `HEADS_DISABLE_USB` | `0` | When `1`, disable automatic USB passthrough and USB cleanup |
| `HEADS_X11_XAUTH` | `0` | When `1`, mount `${HOME}/.Xauthority` into the container (force usage even when a programmatic Xauthority would otherwise be created) |
| `HEADS_SKIP_DOCKER_REBUILD` | `0` | When `1`, skip rebuilding the local Docker image when `flake.nix`/`flake.lock` are dirty |
| `HEADS_AUTO_INSTALL_NIX` | `0` | When `1`, automatically attempt single-user Nix install if `nix` is missing (suppresses prompt) |
| `HEADS_AUTO_ENABLE_FLAKES` | `0` | When `1`, automatically enable flakes by writing to `$HOME/.config/nix/nix.conf` (suppresses prompt) |
| `HEADS_MIN_DISK_GB` | `50` | Minimum free disk in GB required on `/nix` or `/` before attempting rebuild |
| `HEADS_SKIP_DISK_CHECK` | `0` | When `1`, skip the disk-space preflight check |

Examples
---

- Canonical pinned-image path: `DOCKER_REPRO_DIGEST` environment value or `docker/DOCKER_REPRO_DIGEST` fallback, with the resolved canonical digest checked against `.circleci/config.yml` (no CI output comparison):
  - `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run`
  - `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 PUBKEY_ASC=./qemu_img/public-key.asc USB_TOKEN=Nitrokey3NFC inject_gpg`
  - `HEADS_DISABLE_USB=1 ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 PUBKEY_ASC=./qemu_img/public-key.asc run`
  - `HEADS_X11_XAUTH=1 ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run`

- Local development image (uses locally built `linuxboot/heads:dev-env`):
  - `./docker_local_dev.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2`

- Published latest image (convenience):
  - `./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run`

Inspecting wrapper options
---

`source docker/common.sh && build_docker_opts` displays the options that would
be used, including KVM and X11-related mounts when their host prerequisites are
present.  This is an inspection path, not a QEMU or wrapper behavior test.

Troubleshooting
---

- Reuse provisioned canokey state across QEMU board build dirs:
  - QEMU boards that use the default virtual token store canokey state at `build/x86/<BOARD>/.canokey-file` (from `targets/qemu.mk`: `-device canokey,file=$(build)/$(BOARD)/.canokey-file`).
  - After provisioning via Heads OEM reset/re-ownership in one QEMU board, you can copy that file into another QEMU board build directory to reuse the same virtual smartcard identity/public key material.
  - Example:
    - `cp build/x86/qemu-coreboot-fbwhiptail-tpm2/.canokey-file build/x86/qemu-coreboot-fbwhiptail-tpm2-prod_quiet/.canokey-file`
  - This is useful when troubleshooting TPM workflows while keeping the same token identity across variants.

- TPM2 interaction capture (pcap) for debugging, similar to a bus sniffer:
  - On TPM2 boards, set `export CONFIG_TPM2_CAPTURE_PCAP=y` in the board config.
  - Heads `tpmr` then uses the pcap TCTI and writes captures to `/tmp/tpm0.pcap` inside the booted Heads environment.
  - Save/copy that file from the guest (mount-usb --mode rw) and inspect it with Wireshark to analyze TPM command/response traffic.
  - This is intended for TPM2 boards (for example the `qemu-coreboot-fbwhiptail-tpm2*` targets).

- Quick checks:
  - `echo $DISPLAY` — ensure `DISPLAY` is set on the host.
  - `command -v xauth` — preferred for programmatic Xauthority cookies.
  - `ls -l /dev/kvm` — verify `/dev/kvm` exists and is accessible.
  - `groups | grep -q kvm` — confirm your user is in a group with access to KVM (or run with appropriate privileges).
  - `source docker/common.sh && build_docker_opts` — inspect the options the wrapper will use without launching Docker.
- GUI issues: prefer installing `xauth` on the host so the wrappers can create a safe programmatic Xauthority file. As a last resort you can run `xhost +SI:localuser:root` (less secure).
- USB/GPG cleanup: if the cleanup is refusing to run due to non-interactive sudo, run the kill steps manually or set `HEADS_DISABLE_USB=1` to skip automatic cleanup.

Notes
---
- Ensure you have an X server available on the host; the wrappers forward `DISPLAY` automatically.
- If KVM is available but `/dev/kvm` is missing, load kernel modules (e.g., `kvm`, `kvm_intel`, `kvm_amd`) so `/dev/kvm` appears.