Rework workshop first boot

[jonathan-conder]

Description

LXD doesn't support SFTP for stopped VMs, and the implementation for containers is buggy, so the aim of this PR is to replace SFTP with other mechanisms, namely cloud-init and metadata templates.

Behaviour changes

The main behaviour change is that we no longer clean up snapshots as if preparing a new base image. It's technically possible to do this by mounting the snapshot's block device inside the still-running workshop, but we managed to find solutions that work a bit better.

To ensure uniqueness, we now set /etc/machine-id to the LXD config option volatile.uuid. Whatever file ends up in the snapshot is irrelevant. A bonus is that the machine-id is now preserved on refresh, so we can resume using it as the DHCP client identifier.

Another problem with skipping cleanup is that cloud-init changes its behaviour based on whether a workshop's instance ID matches the one stored in the snapshot. Since cloud-init is intended for initial VM setup, we decided to just disable it after the first boot (i.e. when launching or refreshing from a base image).

The main thing cloud-init was responsible for on subsequent boots was generating SSH host keys. Workshop is now responsible for this, which means the keys are preserved on refresh. Even better, we create a signed certificate for each workshop, so users can skip the known_hosts prompt and enable passwordless login by adding this line to their SSH config:

Include /var/snap/workshop/current/ssh/<UID>/config

Workshop format changes

Templates are applied before containers start, and very early in the boot for VMs, so we use them to create:

• /var/lib/workshop/run: needs to exist before LXD initializes the workshop.socket proxy device.
• /etc/cloud/cloud.cfg.d/90_workshop.cfg: needs to exist before cloud-init starts; used to stop cloud-init touching the hostname and bypass netplan in favour of systemd-networkd.
• /etc/machine-id: needs to be set before systemd starts (or passed in as a command-line or kernel parameter); for VMs we'll use a different mechanism based on SMBIOS metadata and GRUB scripts.

Since cloud-init only runs once now, but LXD templates are applied on instance copy, we also use them to create:

• /etc/systemd/network/10-cloud-init-eth0.network.d/workshop.conf: configures workshop-specific networking and settings that cloud-init's network-config doesn't support.
• /etc/ssh/ssh_*_key*: generated on launch, preserved on refresh.

For late-boot and workshop-agnostic config, we use cloud-init to:

• Fixup the permissions of the SSH private host key (hopefully we can remove this in a future LXD version).
• Disable cloud-init (so it only runs when constructing a workshop from a base image).
• Prevent cloud-init from overwriting Workshop's SSH keys.
• Configure the SSH daemon.

Minor changes

• Removes old nvidia.runtime settings from the snapshot tests (with CDI we no longer use these).
• Removes support for old workshop formats (prevents restore or launch|refresh --continue).
• Shows dev-42424242.wp in workshop info if even dev.42424242.wp is unavailable.
• Switches from user.* LXD options to cloud-init.*.
• Removes udisks2 workaround that used to be required to fix suspend.

Self-review quick check

• Make decisions that cost a lot to reverse explicit in the PR description.
• Avoid nested conditions.
• Delete dead code and redundant comments.
• Normalise symmetries by sticking to doing identical things identically.

// one way to handle errors
if err := f(); err != nil {
   ...
}

// one way to handle multiple returns
val, err := f()
if err != nil {
   ...
}
...

• Check that coupled code elements, files, and directories are adjacent. For example, test data is stored as close as possible to a test.
• Put variable declaration and initialisation together.
• Divide large expressions into digestable and self-explanatory ones. Use multiple variables if required.
• Put a blank line between two logically different chunks of code.
• Follow the style guide for new error messages.

Docs

Procedure:

• I have checked and added or updated relevant documentation.
• I have checked and added or updated relevant release notes.
• I have included the technical author in the review.

Content:

• Headings and titles accurately describe the content.
• New and updated pages include correct metadata.
• Documentation tests are added or updated where applicable (for tutorial/ and how-to/ sections).
• Documentation follows the style guide.
• If needed, docs/.coverage.yaml updated, coverage tags added (.. artefact).

---

Or:

• I confirm the PR has no implications for documentation.

[Copilot]

Pull request overview

This PR reworks Workshop “first boot” behavior for LXD-backed workshops by moving more instance initialization into LXD templates and cloud-init config, adding certificate-based SSH key handling, and bumping the snapshot/format revision to reflect the new behavior.

Changes:

• Add LXD instance templates for hostname, machine-id, networkd drop-ins, and SSH host key/cert/CA material; update LXD backend launch/rebuild paths accordingly.
• Update snapshot/copy behavior and integration tests to match the new format (including format revision bump) and new “unique vs shared” config semantics.
• Add snap hook logic and system rules to keep Workshop/LXD-managed networking devices unmanaged by NetworkManager/systemd-networkd; add an integration task for sshutil coverage.

Reviewed changes

Copilot reviewed 34 out of 35 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/integration/ssh/task.yaml	Adds a Spread task to run sshutil integration tests with coverage artifacts.
snap/local/lib/85-workshop-unmanaged.rules	Adds udev rule to mark Workshop bridge devices unmanaged.
snap/local/lib/85-workshop-unmanaged.conf	Adds NetworkManager unmanaged-device config for veth devices (with eth* exception).
snap/local/hooks/remove	Removes installed udev/NM config on snap removal and reloads services.
snap/local/hooks/configure	Installs udev/NM config on configure and reloads services.
internal/workshop/lxd/tests/integration/workshop_test.go	Updates “include when copying” logic to exclude new SSH key config namespace.
internal/workshop/lxd/tests/integration/snapshot-format.yaml	Updates expected snapshot-format metadata/config keys for cloud-init user-data changes.
internal/workshop/lxd/tests/integration/snapshot_test.go	Updates snapshot diff expectations (machine-id/SSH key preservation and new network config path).
internal/workshop/lxd/templates/workshop.socket.untrusted.tpl	Template plumbing for pre-creating the untrusted socket placeholder.
internal/workshop/lxd/templates/ssh_host_ed25519_key.tpl	Adds template for SSH host private key from instance config.
internal/workshop/lxd/templates/ssh_host_ed25519_key.pub.tpl	Adds template for SSH host public key from instance config.
internal/workshop/lxd/templates/ssh_host_ed25519_key-cert.pub.tpl	Adds template for SSH host certificate from instance config.
internal/workshop/lxd/templates/ssh_ca_ed25519_key.pub.tpl	Adds template for SSH user CA public key from instance config.
internal/workshop/lxd/templates/machine-id.tpl	Adds template to derive machine-id from volatile UUID without dashes.
internal/workshop/lxd/templates/hostname.tpl	Adds template for /etc/hostname based on Workshop name.
internal/workshop/lxd/templates/eth0.network.tpl	Adds template for networkd DHCP and domain settings.
internal/workshop/lxd/templates/cloud.cfg.tpl	Adds cloud-init config to force networkd renderer/activator and preserve hostname.
internal/workshop/lxd/lxd_backend.go	Major rework: embeds templates, installs instance metadata templates, generates/persists SSH CA + per-instance host keys/certs, updates cloud-init user-data key.
internal/workshop/lxd/lxd_backend_snapshots.go	Updates snapshot/copy merge semantics, treats new SSH config keys as “unique”, bumps format revision to 5.
internal/workshop/lxd/lxd_backend_sdk.go	Adjusts SDK install behavior to only pre-create mount target when instance is running.
internal/workshop/fakebackend/backend.go	Adds LaunchOrRebuild callback hook and makes SetFormatRevision restoreable for tests.
internal/sshutil/sshutil.go	Introduces SSH key/cert helper package (ed25519 + SSH certificates).
internal/sshutil/sshutil_test.go	Adds integration-tagged tests validating generated keys and signed certs via ssh-keygen.
internal/overlord/workshopstate/manifest.go	Blocks restore/refresh for workshops with incompatible format vs current backend.
internal/overlord/workshopstate/manifest_test.go	Updates tests for restoreable SetFormatRevision and adds restore-format incompatibility test.
internal/overlord/workshopstate/manager.go	Exposes backend format revision via WorkshopManager.
internal/overlord/workshopstate/handlers.go	Removes explicit creation of workshop run dir on base launch path.
internal/overlord/workshopstate/handlers_test.go	Updates cleanup tests to inject errors via LaunchOrRebuild callback instead of WorkshopFs callback.
internal/overlord/sdkstate/handlers_test.go	Updates ResumeAfterWait calls to pass format revision.
internal/overlord/conflict/conflict.go	Extends ResumeAfterWait to require format compatibility when continuing launch/refresh changes.
internal/overlord/conflict/conflict_test.go	Updates ResumeAfterWait tests and adds incompatible-format resume coverage.
internal/dirs/dirs.go	Adds WorkshopSSHDir under the root dir for per-user SSH material.
internal/daemon/snapshot-ingredients.yaml	Updates snapshot ingredients schema to include the new fake backend callback field.
internal/daemon/snapshot-format.yaml	Updates expected snapshot-format outputs (fs-calls and removed run dir).
internal/daemon/api_workshops.go	Passes current format revision into ResumeAfterWait when resuming changes.