docs: DPU CA configuration options

chet · chet · commit 30d4b5228275 · 2026-07-17T12:22:56.000-07:00
Operators can choose how DPUs receive the CA bundle used to authenticate NICo. This collects every supported deployment path and the safe migration steps in one place. Primary callouts are: - Documents the backward-compatible legacy download, site-specific embedded BFB, and operator-mounted bundle modes. - Shows Helm, environment-variable, site configuration, and DPF examples. - Explains staged rollout, root rotation, certificate-chain verification, and the remaining boot-chain trust boundary. This supports #3571 Signed-off-by: Chet Nichols III <chetn@nvidia.com>
diff --git a/book/src/configuration/configurability.md b/book/src/configuration/configurability.md
@@ -415,12 +415,182 @@ overrides. Key fields:
 
 - `default_dpu_agent_version` — DPU agent version installed during provisioning.
 - `bfb_image_path` — BFB (Bluefield boot image) location served to DPUs.
+- `bootstrap_ca_source`: Trust-anchor source for non-DPF DPU provisioning.
 - `dpu_ipmi_tool_impl` (top-level) — `"prod"` for real IPMI; `"fake"` for dev clusters with simulated DPUs.
 - `dpu_ipmi_reboot_attempts` (top-level) — retry budget for IPMI reboot ops.
 
 See [`crates/api-core/src/cfg/README.md` → DpuConfig](../../../crates/api-core/src/cfg/README.md#dpuconfig)
 for the full field list.
 
+#### DPU Bootstrap CA Trust
+
+For non-DPF provisioning, select the source in site config:
+
+```toml
+[dpu_config]
+bootstrap_ca_source = "embedded" # legacy_download | embedded | mounted
+```
+
+The following table describes the available sources:
+
+| Source | Behavior |
+|--------|----------|
+| `legacy_download` | Default when the field is omitted. Preserves the historical download and response handling from `nico-pxe` during boot. It does not authenticate or locally validate the response. |
+| `embedded` | Uses a site-specific CA bundle staged at `/opt/forge/embedded_forge_root.pem` in the DPU BFB from an explicit `BOOTSTRAP_CA_PATH`. Generic builds do not include this payload. A missing or invalid bundle stops provisioning. There is no download fallback. |
+| `mounted` | Uses an operator-managed bundle at `/opt/forge/forge_root.pem`. The provisioning environment must populate the path because the bare-metal flow does not create the mount. A missing or invalid bundle stops provisioning. There is no download fallback. |
+
+##### Operator Configuration Examples
+
+The following examples are independent configurations. Do not combine them.
+Omit the field to preserve the current behavior, or declare the legacy
+behavior explicitly:
+
+```toml
+[dpu_config]
+bootstrap_ca_source = "legacy_download"
+```
+
+Legacy-download sites can serve a stable operator-owned root bundle without
+changing the DPU policy. Reference an existing ConfigMap in the `nico-pxe`
+release namespace. In the NICo umbrella chart's values, use:
+
+```yaml
+nico-pxe:
+  bootstrapRootCa:
+    configMapName: forge-root-ca
+    key: ca.crt
+```
+
+Or reference an existing Secret when that matches the site's distribution
+workflow:
+
+```yaml
+nico-pxe:
+  bootstrapRootCa:
+    secretName: forge-root-ca
+    key: ca.crt
+```
+
+When installing `helm/charts/nico-pxe` directly, omit the `nico-pxe` umbrella
+key. The same top-level shape accepts `secretName` instead of
+`configMapName`:
+
+```yaml
+bootstrapRootCa:
+  configMapName: forge-root-ca
+  key: ca.crt
+```
+
+For a non-Helm `nico-pxe` deployment, point the binary directly at the served
+bundle. The file must already exist in the container, for example through a
+read-only mount, and the launcher must export the variable. Omitting it retains
+the `FORGE_ROOT_CAFILE_PATH` fallback:
+
+```bash
+export FORGE_BOOTSTRAP_ROOT_CAFILE_PATH=/etc/nico/bootstrap/roots.pem
+```
+
+For a non-DPF site that publishes its own BFB, supply the bundle during the BFB
+build and then select the embedded source in NICo site config:
+
+```bash
+BOOTSTRAP_CA_PATH=/secure/site/forge-roots.pem \
+  cargo make --cwd pxe build-boot-artifacts-bfb
+```
+
+```toml
+[dpu_config]
+bootstrap_ca_source = "embedded"
+```
+
+For a non-DPF provisioning environment that installs the bundle itself, place
+it at `/opt/forge/forge_root.pem` and select:
+
+```toml
+[dpu_config]
+bootstrap_ca_source = "mounted"
+```
+
+DPF can retain the download while overriding the complete endpoint URL:
+
+```toml
+[dpf.dpu_agent_bootstrap_ca]
+source = "legacy_download"
+url = "http://site-pxe.example.com/api/v0/tls/root_ca"
+```
+
+Or DPF can install a projected Secret:
+
+```toml
+[dpf.dpu_agent_bootstrap_ca]
+source = "mounted"
+object_kind = "secret"
+name = "nico-bootstrap-ca-v1"
+key = "ca.crt"
+```
+
+The equivalent ConfigMap form is useful when the object is created directly
+in every target DPU cluster:
+
+```toml
+[dpf.dpu_agent_bootstrap_ca]
+source = "mounted"
+object_kind = "config_map"
+name = "nico-bootstrap-ca-v1"
+key = "ca.crt"
+```
+
+This policy is attached only to DPU provisioning instructions. Host Scout
+boots do not receive `[dpu_config].bootstrap_ca_source` and retain their
+existing trust behavior.
+
+The `nico-pxe` chart can decouple the payload returned by the legacy
+`/api/v0/tls/root_ca` endpoint from the CA used by PXE for its own outbound API
+connection. Use either chart form from
+[Operator Configuration Examples](#operator-configuration-examples).
+
+With neither name set, the chart renders the old deployment and serves the
+PXE workload CA as before. This option changes only the bytes served to legacy
+clients. It does not authenticate their HTTP download. Set at most one of
+`configMapName` and `secretName`. Existing DPUs retain the CA they previously
+installed. Changing the served bundle affects only later downloads unless you
+reprovision or refresh the DPU through another trusted mechanism.
+
+For non-Helm deployments, set `FORGE_BOOTSTRAP_ROOT_CAFILE_PATH` to the PEM
+bundle path in the `nico-pxe` container. If it is unset, `nico-pxe` serves the
+file named by `FORGE_ROOT_CAFILE_PATH`, preserving the historical behavior.
+
+Build a site-specific BFB with `BOOTSTRAP_CA_PATH` pointing to the desired PEM
+bundle before selecting `embedded`. The build provides no repository or
+developer-certificate fallback. Without the explicit input, the artifact has
+no dedicated embedded trust anchor. Existing legacy artifact inputs are
+unchanged. The embedded source lives at
+`/opt/forge/embedded_forge_root.pem`, separately from the final
+`/opt/forge/forge_root.pem` path used by `mounted`, so selecting one mode cannot
+silently consume material intended for the other. Roll out the compatible code
+and site-specific artifacts first. Then change site configuration and
+reprovision non-DPF DPUs. During root rotation, use a bundle containing both old
+and new roots until the fleet has moved.
+
+For DPF, configure `[dpf.dpu_agent_bootstrap_ca]` instead. Refer to
+[DPU Agent Bootstrap CA](../../../docs/manuals/dpf.md#dpu-agent-bootstrap-ca)
+for deployment and rotation instructions. DPF configuration changes require a
+`nico-api` restart. The shared published DPU agent image does not embed a site
+trust anchor. DPF supports the compatible legacy download or an
+operator-managed Secret or ConfigMap mounted when the init container starts.
+
+When pinning a root, verify the NICo API presents its intermediate certificate
+with each leaf. Clients cannot build a chain from a root-only bundle if the
+server omits the intermediate. Pinning a long-lived root lets clients validate
+leaf certificates across intermediate rotations without replacing the bundle.
+The bundle validates the API server certificate. This validation is required
+whether the DPU presents a client certificate for mutual TLS. It does not
+authenticate the preceding DHCP, DNS, iPXE, and user-data delivery chain.
+Embedded mode also requires an
+integrity-protected artifact distribution and boot chain, such as verified
+signatures and Secure Boot. Otherwise, an attacker can replace the image and
+its CA together.
+
 ### BIOS profiles — `bios_profiles`, `selected_profile`
 
 `bios_profiles` is a nested map keyed by vendor then model that defines
diff --git a/crates/api-core/src/cfg/README.md b/crates/api-core/src/cfg/README.md
@@ -440,6 +440,7 @@ Extends `StateControllerConfig` with:
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
+| `bootstrap_ca_source` | `BootstrapCaSource` | `legacy_download` | How non-DPF DPUs obtain the API trust anchor: `legacy_download`, `embedded`, or `mounted`. Omitting the field preserves the historical PXE download. The field is not sent to host Scout boots. Non-network modes do not fall back to downloading. |
 | `dpu_nic_firmware_initial_update_enabled` | `bool` | `false` | Enable DPU NIC firmware updates on initial discovery. |
 | `dpu_nic_firmware_reprovision_update_enabled` | `bool` | `true` | Enable DPU NIC firmware updates on reprovisioning. |
 | `dpu_models` | `HashMap<String, Firmware>` | *(BF2+BF3 defaults)* | DPU model firmware definitions. |
@@ -448,6 +449,19 @@ Extends `StateControllerConfig` with:
 | `num_of_vfs` | `u32` | `16` | Number of VFs configured per DPU PF during BlueField provisioning. Max `126`. |
 | `restart_ovs_on_use_admin_network_change` | `bool` | `false` | Restart OVS on DPU-OS agents when host `use_admin_network` changes. Containerized agents skip the local service restart and still ACK the network config. |
 
+To use `embedded`, build a site-specific BFB with an explicit
+`BOOTSTRAP_CA_PATH`. The build provides no repository or default CA fallback
+for the dedicated embedded payload. Existing legacy artifact inputs remain
+unchanged. It stages the source at `/opt/forge/embedded_forge_root.pem`.
+
+This path is separate from `/opt/forge/forge_root.pem`, which the provisioning
+environment must populate for `mounted`. NICo does not create that mount. Both
+modes fail closed when their own bundle is absent or invalid. Changing this
+setting affects the next DPU network boot or reprovisioning. It does not affect
+host Scout boots or rewrite installed DPUs in place. The selected CA validates
+the NICo API server certificate. This validation remains necessary even when
+client-certificate authentication is not used.
+
 ### `NetworkSecurityGroupConfig`
 
 | Field | Type | Default | Description |
@@ -539,11 +553,56 @@ events, so consumers handle them identically.
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
 | `enabled` | `bool` | `false` | Enable DPF Kubernetes deployment. |
-| `services` | `Option<Vec<DpfServiceConfig>>` | — | Additional Helm services. |
+| `dpu_agent_bootstrap_ca` | `DpfDpuAgentBootstrapCa` | `legacy_download` | Bootstrap trust for the containerized DPU agent. Supports `legacy_download` and `mounted`, as described in the following examples. |
+| `services` | `Box<DpfMandatoryServicesConfig>` | built-in mandatory-service defaults | Helm chart, image, and pull-secret settings for the six mandatory DPF services. |
 | `docker_image_pull_secret` | `Option<String>` | — | Override for the Kubernetes `imagePullSecrets` entry used to pull mandatory-service images (applied to every mandatory service except `dts` and `doca_hbn`). |
 | `proxy` | `Option<DpfProxyDetails>` | — | Proxy configuration for the DPU. When set, containerd on the DPU routes outbound HTTPS traffic through it. |
 | `deployments` | `DpfDeploymentsConfig` | *(default)* | Per-generation DPUDeployment configurations. BF3 is always present with defaults; BF4Generic is opt-in via `[dpf.deployments.bf4_generic]`. |
 
+Omitting `[dpf.dpu_agent_bootstrap_ca]` preserves the historical download URL.
+Use the following configuration to retain download mode while overriding the
+complete endpoint URL:
+
+```toml
+[dpf.dpu_agent_bootstrap_ca]
+source = "legacy_download"
+# Optional full endpoint URL. Omit to use the agent default.
+url = "http://carbide-pxe.forge/api/v0/tls/root_ca"
+```
+
+Use the following configuration to project an existing Secret into the DPU
+agent init container:
+
+```toml
+[dpf.dpu_agent_bootstrap_ca]
+source = "mounted"
+object_kind = "secret"
+name = "nico-bootstrap-ca-v1"
+key = "ca.crt"
+```
+
+Use the following configuration for a ConfigMap that already exists in every
+target DPU cluster:
+
+```toml
+[dpf.dpu_agent_bootstrap_ca]
+source = "mounted"
+object_kind = "config_map"
+name = "nico-bootstrap-ca-v1"
+key = "ca.crt"
+```
+
+The URL override changes routing, not the initial trust model. An HTTPS URL is
+authenticated only when its server certificate chains to a root already
+trusted by the shared dpu-agent image.
+
+Mounted mode never falls back to the legacy download. The shared published
+dpu-agent image does not embed a site-specific trust anchor. A mounted
+ConfigMap must already exist in the DPU cluster. A suitably labeled Secret can
+be propagated there by DPF. Refer to
+[DPU Agent Bootstrap CA](../../../../docs/manuals/dpf.md#dpu-agent-bootstrap-ca)
+for deployment and rotation instructions.
+
 ### `RmsConfig`
 
 | Field | Type | Default | Description |
diff --git a/docs/bootable_artifacts.md b/docs/bootable_artifacts.md
@@ -8,7 +8,57 @@ Install 'mkosi' and 'debootstrap' from the repository -- for Debian it was
 sudo apt install mkosi debootstrap
 ```
 
-### 2. Build IPXE image
+### 2. Select the Bootstrap CA Bundle
+
+Site-specific BlueField bootstream (BFB) builds accept an operator-supplied
+certificate authority (CA) bundle in Privacy-Enhanced Mail (PEM) format through
+`BOOTSTRAP_CA_PATH`:
+
+```bash
+export BOOTSTRAP_CA_PATH=/absolute/path/to/site-bootstrap-roots.pem
+```
+
+Build the production BFB from a supported environment with the existing
+aarch64 cross-build, BFB tooling, registry, and network prerequisites:
+
+```bash
+cargo make --cwd pxe build-boot-artifacts-bfb
+```
+
+`embedded` mode is available only when this variable is supplied explicitly.
+There is no fallback to `FORGE_CA_PATH`, a repository certificate, or a
+developer certificate. When it is absent, the dedicated embedded payload is
+removed, while existing legacy artifact inputs remain unchanged. Selecting
+`embedded` with such an artifact fails closed instead of downloading a CA.
+
+The artifact stores this build-time input at the dedicated embedded source
+`/opt/forge/embedded_forge_root.pem`. At DPU boot, embedded mode copies it into
+the final `/opt/forge/forge_root.pem` location. This source is intentionally
+distinct from `mounted` mode, which expects the provisioning environment to
+populate the final path directly, so one mode cannot silently use material
+intended for the other.
+
+The `[dpu_config].bootstrap_ca_source` policy is sent only on DPU provisioning
+paths. Host Scout boots do not consume it. Use the same bundle for every BFB
+variant deployed at a site.
+
+For a root rotation, first build a bundle containing both the old and new
+roots, publish and deploy those artifacts, reprovision the fleet, and only then
+build a bundle without the old root. A non-DPF `mounted` deployment instead
+expects the provisioning environment to place the operator-managed bundle at
+`/opt/forge/forge_root.pem`. NICo does not create that mount.
+
+Upgrade the NICo control plane and publish compatible artifacts before enabling
+`embedded` or `mounted`. Older artifacts support only the legacy download.
+Verify that the NICo API serves the intermediate certificate with its leaf when
+the artifact pins only the root. The bundle performs TLS server certificate
+validation whether the agent uses a client certificate for mutual TLS.
+
+Embedding moves trust into the artifact. Protect artifact publication and
+distribution with verified signatures. Enforce Secure Boot or an equivalent
+chain of trust before treating the embedded CA as a pinned anchor.
+
+### 3. Build iPXE Image
 
 Run
 
diff --git a/docs/dpu-management/dpu_configuration.md b/docs/dpu-management/dpu_configuration.md
@@ -46,6 +46,58 @@ sequenceDiagram
     end
 ```
 
+## Bootstrap CA Trust
+
+Before a DPU agent can use the configuration APIs, it needs a certificate
+authority (CA) bundle to authenticate the NICo API. Configure the bundle source
+for non-DPF provisioning in the site configuration:
+
+```toml
+[dpu_config]
+bootstrap_ca_source = "legacy_download" # legacy_download | embedded | mounted
+```
+
+If this field is omitted, NICo preserves the historical behavior: the booting
+DPU downloads `/api/v0/tls/root_ca` from `nico-pxe`. This supports rolling
+upgrades but the download remains dependent on unauthenticated DHCP, DNS, and
+HTTP. `embedded` uses a site-specific bundle staged into the DPU BFB only when
+its build is given an explicit `BOOTSTRAP_CA_PATH`. There is no repository or
+default fallback for the dedicated embedded payload. Existing legacy artifact
+inputs remain unchanged. `mounted` instead expects the provisioning environment
+to place an operator-managed bundle at `/opt/forge/forge_root.pem`. NICo does
+not create that mount. The embedded
+source at `/opt/forge/embedded_forge_root.pem` and mounted final path are
+distinct. Both non-network modes fail closed when their own bundle is missing
+or invalid and never fall back to the download.
+
+NICo includes this setting only in DPU provisioning instructions. It does not
+change host Scout boot behavior.
+
+Upgrade NICo and publish compatible boot artifacts before you change the
+setting. To switch an installed non-DPF DPU, reprovision it or use another
+trusted mechanism to install the bundle. Changing site configuration alone
+does not rewrite its filesystem. For root rotation, build or mount an overlap
+bundle containing the old and new roots. Migrate the fleet, then remove the old
+root.
+
+Containerized DPF agents use the separate
+`[dpf.dpu_agent_bootstrap_ca]` policy and apply it when their init container
+starts. Refer to
+[DPU Agent Bootstrap CA](../manuals/dpf.md#dpu-agent-bootstrap-ca) for the
+legacy-download and mounted-object forms. The shared published DPF
+dpu-agent image does not embed a site trust anchor.
+
+When pinning a root, verify that the NICo API sends the issuing intermediate
+certificate with its leaf certificate. Selecting a stable root establishes the
+intended trust anchor. With the complete chain, clients can validate leaf
+certificates across intermediate rotations without replacing the bundle. This
+is server certificate validation and remains required whether
+client-certificate authentication is enabled. It does not authenticate the
+earlier DHCP, DNS, iPXE, and user-data boot chain. Embedded mode also depends on
+an authenticated artifact and boot chain, such as verified image signatures
+plus Secure Boot. Otherwise, an attacker can replace both the artifact and its
+CA.
+
 ## Configuration Versioning
 
 NICo uses versioned immutable configuration data in order to detect whether any intended changes have not yet been deployed:
diff --git a/docs/manuals/dpf.md b/docs/manuals/dpf.md
diff --git a/docs/provisioning/ip-and-network-configuration.md b/docs/provisioning/ip-and-network-configuration.md
