diff --git a/book/src/configuration/configurability.md b/book/src/configuration/configurability.md index 999ab5e3a6..fbc30753b7 100644 --- a/book/src/configuration/configurability.md +++ b/book/src/configuration/configurability.md @@ -415,12 +415,188 @@ overrides. Key fields: - `default_dpu_agent_version` — DPU agent version installed during provisioning. - `bfb_image_path` — BFB (Bluefield boot image) location served to DPUs. +- `bootstrap_ca_source`: Trust-anchor source for non-DPF DPU provisioning. - `dpu_ipmi_tool_impl` (top-level) — `"prod"` for real IPMI; `"fake"` for dev clusters with simulated DPUs. - `dpu_ipmi_reboot_attempts` (top-level) — retry budget for IPMI reboot ops. See [`crates/api-core/src/cfg/README.md` → DpuConfig](../../../crates/api-core/src/cfg/README.md#dpuconfig) for the full field list. +#### DPU Bootstrap CA Trust + +For non-DPF provisioning, select the source in site config: + +```toml +[dpu_config] +bootstrap_ca_source = "embedded" # legacy_download | embedded | mounted +``` + +The following table describes the available sources: + +| Source | Behavior | +|--------|----------| +| `legacy_download` | Default when the field is omitted. Preserves the historical download and response handling from `nico-pxe` during boot. It does not authenticate or locally validate the response. | +| `embedded` | Uses a site-specific CA bundle staged at `/opt/forge/embedded_forge_root.pem` in the DPU BFB from an explicit `BOOTSTRAP_CA_PATH`. Generic builds do not include this payload. A missing or invalid bundle stops provisioning. There is no download fallback. | +| `mounted` | Uses an operator-managed bundle at `/opt/forge/forge_root.pem`. The provisioning environment must populate the path because the bare-metal flow does not create the mount. A missing or invalid bundle stops provisioning. There is no download fallback. | + +##### Operator Configuration Examples + +The following examples are independent configurations. Do not combine them. +Omit the field to preserve the current behavior, or declare the legacy +behavior explicitly: + +```toml +[dpu_config] +bootstrap_ca_source = "legacy_download" +``` + +Legacy-download sites can serve a stable operator-owned root bundle without +changing the DPU policy. Reference an existing ConfigMap in the `nico-pxe` +release namespace. In the NICo umbrella chart's values, use: + +```yaml +nico-pxe: + bootstrapRootCa: + configMapName: forge-root-ca + key: ca.crt +``` + +Or reference an existing Secret when that matches the site's distribution +workflow: + +```yaml +nico-pxe: + bootstrapRootCa: + secretName: forge-root-ca + key: ca.crt +``` + +When installing `helm/charts/nico-pxe` directly, omit the `nico-pxe` umbrella +key. The same top-level shape accepts `secretName` instead of +`configMapName`: + +```yaml +bootstrapRootCa: + configMapName: forge-root-ca + key: ca.crt +``` + +For a non-Helm `nico-pxe` deployment, point the binary directly at the served +bundle. The file must already exist in the container, for example through a +read-only mount, and the launcher must export the variable. Omitting it retains +the `FORGE_ROOT_CAFILE_PATH` fallback: + +```bash +export FORGE_BOOTSTRAP_ROOT_CAFILE_PATH=/etc/nico/bootstrap/roots.pem +``` + +For a non-DPF site that publishes its own BFB, supply the bundle during the BFB +build and then select the embedded source in NICo site config: + +```bash +BOOTSTRAP_CA_PATH=/secure/site/forge-roots.pem \ + cargo make --cwd pxe build-boot-artifacts-bfb +``` + +```toml +[dpu_config] +bootstrap_ca_source = "embedded" +``` + +For a non-DPF provisioning environment that installs the bundle itself, place +it at `/opt/forge/forge_root.pem` and select: + +```toml +[dpu_config] +bootstrap_ca_source = "mounted" +``` + +DPF can retain the download while overriding the complete endpoint URL: + +```toml +[dpf.dpu_agent_bootstrap_ca] +source = "legacy_download" +url = "http://site-pxe.example.com/api/v0/tls/root_ca" +``` + +Or DPF can install a projected Secret: + +```toml +[dpf.dpu_agent_bootstrap_ca] +source = "mounted" +object_kind = "secret" +name = "nico-bootstrap-ca-v1" +key = "ca.crt" +``` + +The equivalent ConfigMap form is useful when the object is created directly +in every target DPU cluster: + +```toml +[dpf.dpu_agent_bootstrap_ca] +source = "mounted" +object_kind = "config_map" +name = "nico-bootstrap-ca-v1" +key = "ca.crt" +``` + +This policy is attached only to DPU provisioning instructions. Host Scout +boots do not receive `[dpu_config].bootstrap_ca_source` and retain their +existing trust behavior. + +The `nico-pxe` chart can decouple the payload returned by the legacy +`/api/v0/tls/root_ca` endpoint from the CA used by PXE for its own outbound API +connection. Use either chart form from +[Operator Configuration Examples](#operator-configuration-examples). + +With neither name set, the chart renders the old deployment and serves the +PXE workload CA as before. This option changes only the bytes served to legacy +clients. It does not authenticate their HTTP download. Set at most one of +`configMapName` and `secretName`. Existing DPUs retain the CA they previously +installed. Changing the served bundle affects only later downloads unless you +reprovision or refresh the DPU through another trusted mechanism. + +For non-Helm deployments, set `FORGE_BOOTSTRAP_ROOT_CAFILE_PATH` to the PEM +bundle path in the `nico-pxe` container. If it is unset, `nico-pxe` serves the +file named by `FORGE_ROOT_CAFILE_PATH`, preserving the historical behavior. + +Build a site-specific BFB with `BOOTSTRAP_CA_PATH` pointing to the desired PEM +bundle before selecting `embedded`. The build provides no repository or +developer-certificate fallback. Without the explicit input, the artifact has +no dedicated embedded trust anchor. Existing legacy artifact inputs are +unchanged. The embedded source lives at +`/opt/forge/embedded_forge_root.pem`, separately from the final +`/opt/forge/forge_root.pem` path used by `mounted`, so selecting one mode cannot +silently consume material intended for the other. Roll out the compatible code +and site-specific artifacts first. Then change site configuration and +reprovision non-DPF DPUs. During root rotation, publish an overlap bundle +containing both old and new roots and reprovision every DPU. Verify that every +DPU installed the overlap bundle and can authenticate the NICo API before +rotating the API server chain to the new root. Verify authentication again, +then publish a bundle without the old root, reprovision and verify the fleet, +and retire the old root only after that rollout succeeds. + +For DPF, configure `[dpf.dpu_agent_bootstrap_ca]` instead. Refer to +[DPU Agent Bootstrap CA](../../../docs/manuals/dpf.md#dpu-agent-bootstrap-ca) +for deployment and rotation instructions. DPF configuration changes require a +`carbide-api` restart. The shared published DPU agent image does not embed a site +trust anchor. DPF supports the compatible legacy download or an +operator-managed Secret or ConfigMap mounted when the init container starts. + +When pinning a root, verify the NICo API presents its intermediate certificate +with each leaf. Clients cannot build a chain from a root-only bundle if the +server omits the intermediate. If each replacement intermediate chains to the +pinned root and the server presents the complete chain, clients can validate +leaf certificates across those rotations without replacing the bundle. If an +intermediate chains to a different root, stage and verify an updated root bundle +before rotating the server chain. The bundle validates the API server +certificate. This validation is required whether the DPU presents a client +certificate for mutual TLS. It does not authenticate the preceding DHCP, DNS, +iPXE, and user-data delivery chain. Embedded mode also requires an +integrity-protected artifact distribution and boot chain, such as verified +signatures and Secure Boot. Otherwise, an attacker can replace the image and +its CA together. + ### BIOS profiles — `bios_profiles`, `selected_profile` `bios_profiles` is a nested map keyed by vendor then model that defines diff --git a/docs/bootable_artifacts.md b/docs/bootable_artifacts.md index 919ef99a0f..04516964e7 100644 --- a/docs/bootable_artifacts.md +++ b/docs/bootable_artifacts.md @@ -8,7 +8,63 @@ Install 'mkosi' and 'debootstrap' from the repository -- for Debian it was sudo apt install mkosi debootstrap ``` -### 2. Build IPXE image +### 2. Select the Bootstrap CA Bundle + +Site-specific BlueField bootstream (BFB) builds accept an operator-supplied +certificate authority (CA) bundle in Privacy-Enhanced Mail (PEM) format through +`BOOTSTRAP_CA_PATH`: + +```bash +export BOOTSTRAP_CA_PATH=/absolute/path/to/site-bootstrap-roots.pem +``` + +Build the production BFB from a supported environment with the existing +aarch64 cross-build, BFB tooling, registry, and network prerequisites: + +```bash +cargo make --cwd pxe build-boot-artifacts-bfb +``` + +`embedded` mode is available only when this variable is supplied explicitly. +There is no fallback to `FORGE_CA_PATH`, a repository certificate, or a +developer certificate. When it is absent, the dedicated embedded payload is +removed, while existing legacy artifact inputs remain unchanged. Selecting +`embedded` with such an artifact fails closed instead of downloading a CA. + +The artifact stores this build-time input at the dedicated embedded source +`/opt/forge/embedded_forge_root.pem`. At DPU boot, embedded mode copies it into +the final `/opt/forge/forge_root.pem` location. This source is intentionally +distinct from `mounted` mode, which expects the provisioning environment to +populate the final path directly, so one mode cannot silently use material +intended for the other. + +The `[dpu_config].bootstrap_ca_source` policy is sent only on DPU provisioning +paths. Host Scout boots do not consume it. Use the same bundle for every BFB +variant deployed at a site. + +For a root rotation, first build a bundle containing both the old and new +roots, publish and deploy those artifacts, and reprovision every DPU. Verify +that every DPU installed the overlap bundle at `/opt/forge/forge_root.pem` and +can authenticate the NICo API. Rotate the API server chain to the new root and +verify authentication again while the overlap bundle is installed. Only then +publish artifacts without the old root, reprovision and verify the fleet again, +and retire the old root and artifacts. A non-DPF `mounted` deployment instead +expects the provisioning environment to place the operator-managed bundle at +`/opt/forge/forge_root.pem`. NICo does not create that mount. Apply the same +fleet-wide installation and authentication gates when rotating a mounted +bundle. + +Upgrade the NICo control plane and publish compatible artifacts before enabling +`embedded` or `mounted`. Older artifacts support only the legacy download. +Verify that the NICo API serves the intermediate certificate with its leaf when +the artifact pins only the root. The bundle performs TLS server certificate +validation whether the agent uses a client certificate for mutual TLS. + +Embedding moves trust into the artifact. Protect artifact publication and +distribution with verified signatures. Enforce Secure Boot or an equivalent +chain of trust before treating the embedded CA as a pinned anchor. + +### 3. Build iPXE Image Run diff --git a/docs/dpu-management/dpu_configuration.md b/docs/dpu-management/dpu_configuration.md index bc9428bc76..1b63e898cc 100644 --- a/docs/dpu-management/dpu_configuration.md +++ b/docs/dpu-management/dpu_configuration.md @@ -46,6 +46,63 @@ sequenceDiagram end ``` +## Bootstrap CA Trust + +Before a DPU agent can use the configuration APIs, it needs a certificate +authority (CA) bundle to authenticate the NICo API. Configure the bundle source +for non-DPF provisioning in the site configuration: + +```toml +[dpu_config] +bootstrap_ca_source = "legacy_download" # legacy_download | embedded | mounted +``` + +If this field is omitted, NICo preserves the historical behavior: the booting +DPU downloads `/api/v0/tls/root_ca` from `nico-pxe`. This supports rolling +upgrades but the download remains dependent on unauthenticated DHCP, DNS, and +HTTP. `embedded` uses a site-specific bundle staged into the DPU BFB only when +its build is given an explicit `BOOTSTRAP_CA_PATH`. There is no repository or +default fallback for the dedicated embedded payload. Existing legacy artifact +inputs remain unchanged. `mounted` instead expects the provisioning environment +to place an operator-managed bundle at `/opt/forge/forge_root.pem`. NICo does +not create that mount. The embedded +source at `/opt/forge/embedded_forge_root.pem` and mounted final path are +distinct. Both non-network modes fail closed when their own bundle is missing +or invalid and never fall back to the download. + +NICo includes this setting only in DPU provisioning instructions. It does not +change host Scout boot behavior. + +Upgrade NICo and publish compatible boot artifacts before you change the +setting. To switch an installed non-DPF DPU, reprovision it or use another +trusted mechanism to install the bundle. Changing site configuration alone +does not rewrite its filesystem. For root rotation, build or mount an overlap +bundle containing the old and new roots. Reprovision or refresh every DPU, then +verify that each one installed the overlap bundle at +`/opt/forge/forge_root.pem` and can authenticate the NICo API. Rotate the API +server chain to the new root and verify authentication again. Only then deploy +a bundle without the old root, verify every DPU again, and retire the old root. + +Containerized DPF agents use the separate +`[dpf.dpu_agent_bootstrap_ca]` policy and apply it when their init container +starts. Refer to +[DPU Agent Bootstrap CA](../manuals/dpf.md#dpu-agent-bootstrap-ca) for the +legacy-download and mounted-object forms. The shared published DPF +dpu-agent image does not embed a site trust anchor. + +When pinning a root, verify that the NICo API sends the issuing intermediate +certificate with its leaf certificate. Selecting a stable root establishes the +intended trust anchor. If each replacement intermediate chains to the pinned +root and the server presents the complete chain, clients can validate leaf +certificates across those rotations without replacing the bundle. If an +intermediate chains to a different root, stage and verify an updated bundle +before rotating the server chain. This is server certificate validation and +remains required whether client-certificate authentication is enabled. It does +not authenticate the earlier DHCP, DNS, iPXE, and user-data boot chain. Embedded +mode also depends on an authenticated artifact and boot chain, such as verified +image signatures plus Secure Boot. Otherwise, an attacker can replace both the +artifact and its CA. + ## Configuration Versioning NICo uses versioned immutable configuration data in order to detect whether any intended changes have not yet been deployed: diff --git a/docs/manuals/dpf.md b/docs/manuals/dpf.md index 5fda190983..87aab88917 100644 --- a/docs/manuals/dpf.md +++ b/docs/manuals/dpf.md @@ -634,6 +634,7 @@ Field reference (all under `[dpf]`): | --- | --- | --- | --- | | `enabled` | bool | `false` | Master switch. Must be `true` to use DPF-based provisioning. | | `docker_image_pull_secret` | string | `dpf-pull-secret` | Pull Secret applied to every mandatory service except `dts` and `doca_hbn`. | +| `dpu_agent_bootstrap_ca` | tagged table | `source = "legacy_download"` | Selects legacy download or mounted-object bootstrap trust for the DPU agent. | | `services.` | table | per-service defaults | Helm/image overrides for each mandatory DPUService. | | `deployments.bf3` | table | BF3 defaults | BF3 DPUDeployment config; always active. | | `deployments.bf4_generic` | table | — | BF4 (generic) DPUDeployment config; opt-in, active only when present. | @@ -647,6 +648,109 @@ Notes: to talk to the host cluster are **not** configured here — carbide-api uses its in-cluster ServiceAccount and the fixed `dpf-operator-system` namespace. +#### DPU Agent Bootstrap CA + +The containerized DPU agent has its own bootstrap policy. If the table is +absent, its init container preserves the historical PXE download: + +```toml +[dpf.dpu_agent_bootstrap_ca] +source = "legacy_download" +# Optional. When set, this must be the full endpoint URL, not a PXE base URL. +url = "http://carbide-pxe.forge/api/v0/tls/root_ca" +``` + +The URL override changes where the bundle is downloaded. It does not establish +bootstrap trust by itself. HTTPS authenticates this fetch only when the shared +DPU agent image already trusts the endpoint's certificate chain. + +Use the following configuration to project an operator-managed bundle into the +init container: + +```toml +[dpf.dpu_agent_bootstrap_ca] +source = "mounted" +object_kind = "secret" +name = "nico-bootstrap-ca-v1" +key = "ca.crt" +``` + +Use the following equivalent ConfigMap configuration: + +```toml +[dpf.dpu_agent_bootstrap_ca] +source = "mounted" +object_kind = "config_map" +name = "nico-bootstrap-ca-v1" +key = "ca.crt" +``` + +The shared published DPU agent image does not contain a site-specific trust +anchor, so DPF does not expose an `embedded` source. The `mounted` source +validates and installs the projected bundle. It fails closed when the bundle is +absent or invalid and never falls back to the legacy download. Upgrade the DPU +agent image and NICo API before selecting `mounted`. If the table is absent, +the chart renders the historical `init-container` invocation for rolling +compatibility. Mounted mode requires the DPU agent chart's `certsDir` to remain +at its default `/opt/forge`. The main container uses this fixed CA installation +path too. + +The referenced object must exist in the `dpu-agent` workload namespace in every +target DPU cluster. DPF does not propagate ConfigMaps, so create one in each +cluster. To propagate a Secret from the host cluster, apply DPF's established +label: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: nico-bootstrap-ca-v1 + namespace: dpf-operator-system + labels: + dpu.nvidia.com/image-pull-secret: "" +type: Opaque +stringData: + ca.crt: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- +``` + +The label name is historical. The CA is public and is not an image-pull +credential. Confirm that the Secret has appeared in the target DPU cluster +before enabling `mounted`. + +The NICo API reads changes under `[dpf]` only at startup. Updating only the +contents of an object with the same name does not guarantee a pod restart or a +newly installed CA. Use the following sequence for a mounted root rotation: + +1. Create a new versioned Secret or ConfigMap containing an overlap bundle with + both the old and new roots. +2. Set `[dpf.dpu_agent_bootstrap_ca].name` to the new object name and restart + `carbide-api`. +3. Wait for DPF to reconcile the service template and for every affected DPU + agent pod to roll and run its init container again. +4. Verify that each pod installed the overlap bundle at + `/opt/forge/forge_root.pem` and can authenticate the NICo API certificate + chain. +5. Rotate the NICo API certificate chain to one that terminates at the new root. + While the overlap bundle is installed, verify that every affected DPU agent + can authenticate the new server chain. +6. Create another versioned object without the old root and repeat the + configuration, restart, reconciliation, and verification steps. Remove the + old objects only after that rollout succeeds. + +When pinning a root, verify that the NICo API serves the issuing intermediate +certificate with its leaf. This policy controls trust-anchor selection. If each +replacement intermediate chains to the pinned root and the server presents the +complete chain, clients can validate leaf certificates across those rotations +without replacing the bundle. If an intermediate chains to a different root, +stage and verify an updated root bundle before rotating the server chain. TLS +server certificate validation remains necessary even if the DPU agent no +longer presents a client certificate for mutual TLS. It does not authenticate +the preceding artifact or provisioning chain. Those inputs still require +integrity protection and a trusted boot mechanism such as Secure Boot. + ### 3.6. Mark hosts as DPF-managed in expected machines Whether a given host is provisioned via DPF or via iPXE is decided per host, diff --git a/docs/provisioning/ip-and-network-configuration.md b/docs/provisioning/ip-and-network-configuration.md index e7dd919fea..b680673638 100644 --- a/docs/provisioning/ip-and-network-configuration.md +++ b/docs/provisioning/ip-and-network-configuration.md @@ -291,7 +291,7 @@ The required A records (shown for `.nico`; substitute `.nico` if your binaries u | Hostname | Port | Resolves to | Purpose | Configurable at runtime? | |---|---|---|---|---| | `nico-api.nico` | 443 | `nico-api` external LoadBalancer VIP | NICo gRPC API | Yes — `NICO_API_URL` env var on most clients | -| `nico-pxe.nico` | 80 | `nico-pxe` LoadBalancer VIP | iPXE scripts, cloud-init, internal APT, TLS root CA | **No** — hardcoded in the compiled DPU agent | +| `nico-pxe.nico` | 80 | `nico-pxe` LoadBalancer VIP | iPXE scripts, cloud-init, internal APT, and the legacy bootstrap-CA endpoint | The DNS record remains fixed for general consumers. DPF can separately configure bootstrap CA acquisition through a complete URL override or mounted Secret or ConfigMap. Non-DPF boot instructions include `pxe_uri`. Other consumers retain the compatibility hostname. | | `nico-static-pxe.nico` | 80 | Static PXE asset server VIP | `scout.squashfs`, `scout.efi`, BFB images, and other static boot artifacts | **No** — hardcoded in the host boot scripts that ship inside boot images | | `nico-ntp.nico` | 123 | Operator-supplied NTP server IP(s) — the record points at your existing NTP infrastructure, not a NICo-deployed service | Legacy NTP fallback for DPU agents when `siteConfig.ntp_servers` is empty | **Fallback only** — prefer `siteConfig.ntp_servers`, but keep this DNS record if any deployed agent still relies on it | | `unbound.nico` | 53 | `unbound` LoadBalancer VIP | Recursive DNS resolver | Yes — the resolver address itself is distributed via DHCP option 6 | @@ -301,6 +301,73 @@ One additional `.nico` hostname, `socks.nico`, is hardcoded into the DPU agent a > **Note:** Neither `.nico` nor `.nico` is a publicly registered TLD. Both are used exclusively on the isolated OOB management network. Configure the recursive resolver to treat the chosen TLD as locally authoritative and **not** forward queries to upstream public resolvers. +#### Bootstrap CA Selection and Network Trust + +If you do not configure a bootstrap certificate authority (CA), DPUs retain the +historical behavior and download +`http:///api/v0/tls/root_ca`. You can make the bytes served by that +legacy endpoint independent of PXE's own outbound API trust. Reference an +existing ConfigMap or Secret in the `nico-pxe` Helm release namespace: + +```yaml +nico-pxe: + bootstrapRootCa: + configMapName: forge-root-ca # Set secretName instead for a Secret. + key: ca.crt +``` + +Set either `configMapName` or `secretName`, but not both. If you set neither, +the chart preserves the old PXE deployment and payload path. This option can +serve a stable root instead of a rotating site intermediate. It does not +authenticate a CA fetched over DHCP-directed, DNS-resolved HTTP. Existing DPUs +retain the CA they already installed. A payload change affects only later +downloads unless you reprovision or refresh the DPU through another trusted +mechanism. + +Outside Helm, set `FORGE_BOOTSTRAP_ROOT_CAFILE_PATH` to the PEM bundle path in +the `nico-pxe` container. If you do not set it, `nico-pxe` serves the file named +by `FORGE_ROOT_CAFILE_PATH`, preserving the historical bundle. + +Non-DPF deployments can set `[dpu_config].bootstrap_ca_source` to `embedded` +or `mounted`. That setting applies only to DPU provisioning, not host Scout +boots. Embedded mode requires a site-specific BFB build with an explicit +`BOOTSTRAP_CA_PATH`. There is no default fallback for the dedicated embedded +payload. Existing legacy artifact inputs remain unchanged. Mounted mode +consumes the operator-populated final `/opt/forge/forge_root.pem` path instead +of the distinct embedded `/opt/forge/embedded_forge_root.pem` source. DPF +deployments use +`[dpf.dpu_agent_bootstrap_ca]` and support the legacy download or a mounted +Secret or ConfigMap. In legacy mode, `url` can replace the complete default +endpoint with an HTTP or HTTPS URL. The +[DPU Agent Bootstrap CA](../manuals/dpf.md#dpu-agent-bootstrap-ca) section +provides full URL and mounted-object examples. The shared published DPU agent +image does not embed a site CA. Non-network modes require a valid local bundle +and fail closed without falling back to the download. Upgrade code, images, and +boot artifacts before enabling them. Reprovision non-DPF DPUs. For DPF changes, +restart `carbide-api` and roll the DPU agent pods. + +Before pinning a root, inspect the NICo API certificate chain with the same DNS +name the DPU uses: + +```bash +export NICO_API_HOST='' +openssl s_client -connect "${NICO_API_HOST}:443" \ + -servername "${NICO_API_HOST}" -verify_hostname "${NICO_API_HOST}" \ + -showcerts -verify_return_error \ + -CAfile /path/to/site-bootstrap-roots.pem