From 904bea57e5488a4f8606db2693cab5719d3e0a95 Mon Sep 17 00:00:00 2001 From: Jakhongir Rakhmonov Date: Fri, 17 Jul 2026 15:30:33 +0000 Subject: [PATCH] firmware docs overhaul --- docs/index.yml | 18 +- docs/operations/firmware-updates.md | 601 ++++-------------- .../firmware-updates/configuration.md | 382 +++++++++++ .../firmware-updates/dpu-firmware.md | 248 ++++++++ .../firmware-updates/host-firmware.md | 382 +++++++++++ .../firmware-updates/pre-ingestion.md | 184 ++++++ .../rack-component-firmware.md | 332 ++++++++++ .../scout-firmware-assigned-host-gap.md | 142 +++++ 8 files changed, 1809 insertions(+), 480 deletions(-) create mode 100644 docs/operations/firmware-updates/configuration.md create mode 100644 docs/operations/firmware-updates/dpu-firmware.md create mode 100644 docs/operations/firmware-updates/host-firmware.md create mode 100644 docs/operations/firmware-updates/pre-ingestion.md create mode 100644 docs/operations/firmware-updates/rack-component-firmware.md create mode 100644 docs/operations/scout-firmware-assigned-host-gap.md diff --git a/docs/index.yml b/docs/index.yml index 6ea13bc029..3da684fd4a 100644 --- a/docs/index.yml +++ b/docs/index.yml @@ -152,8 +152,22 @@ navigation: path: manuals/dpu_extension_service.md - page: BlueField DPU Operations path: dpu-management/dpu-operations.md - - page: Firmware Updates - path: operations/firmware-updates.md + - section: Firmware Upgrades + contents: + - page: Overview + path: operations/firmware-updates.md + - page: Configure Firmware Versions + path: operations/firmware-updates/configuration.md + - page: Pre-ingestion Firmware Upgrades + path: operations/firmware-updates/pre-ingestion.md + - page: Managed Host Firmware Upgrades + path: operations/firmware-updates/host-firmware.md + - page: DPU Firmware Upgrades + path: operations/firmware-updates/dpu-firmware.md + - page: Rack and Tray Firmware Upgrades + path: operations/firmware-updates/rack-component-firmware.md + - page: Flow Operation Rules + path: ../rest-api/flow/docs/operation-rules-guide.md - page: Monitoring and Health path: operations/monitoring-health.md - page: Tenant Lifecycle Cleanup diff --git a/docs/operations/firmware-updates.md b/docs/operations/firmware-updates.md index 017d21b5a5..a5ac7df8c6 100644 --- a/docs/operations/firmware-updates.md +++ b/docs/operations/firmware-updates.md @@ -1,479 +1,124 @@ -# Firmware Updates - -This page covers when NICo updates firmware, how automated updates are -scheduled, how to control update windows, and where NICo hands off to -platform-specific firmware procedures. - -Firmware updates run in two operating moments: - -- **Initial ingestion and pre-ingestion.** NICo can update firmware before a - host is fully ingested when the current firmware is too old for safe - discovery, host-to-DPU pairing, or the configured minimum firmware policy. -- **Approved service windows.** After NICo manages a machine, firmware updates - are scheduled during approved maintenance windows and are limited by health - and concurrency policy. - -For related background, see: - -- [Architecture Overview](../architecture/overview.md#machine-update-manager) -- [Managed Host State Diagrams](../architecture/state_machines/managedhost.md) -- [DPU Lifecycle Management](../dpu-management/dpu-lifecycle-management.md#firmware-upgrades) -- [Redfish Workflow](../architecture/redfish_workflow.md) -- [Core Metrics](../observability/core_metrics.md) - -## Workflow Boundaries - -NICo has several firmware paths. Use the path that matches the component being -updated. - -| Scope | NICo path | Use | +# Firmware upgrades + +This guide first gives a high-level view of firmware upgrades in NICo and helps +you choose the workflow that applies to your hardware. The linked workflow +pages then explain their configuration, prerequisites, execution, verification, +and recovery in detail. + +## Overview + +NICo uses a site-wide desired-state process for managed host firmware and DPU +NIC firmware. Operators configure the versions expected at the site. When +automatic updates are enabled, NICo compares those versions with the firmware +reported by managed hardware. Drift makes a device eligible for an upgrade, +which NICo schedules automatically instead of requiring a one-off upgrade +command. + +The drift-driven workflow is: + +```mermaid +flowchart LR + A["Configured host or DPU NIC versions"] --> B["Compare with discovered versions"] + B --> C{"Drift detected?"} + C -- No --> D["No action"] + C -- Yes --> E["Wait until the device is eligible"] + E --> F["Upgrade or reprovision"] + F --> G["Rediscover and verify"] +``` + +Detecting drift does not mean the upgrade starts immediately. NICo may wait for +the machine to enter an eligible state, an update window to open, or site-wide +concurrency limits to allow more work. The responsible controller then performs +the update or reprovisioning and verifies the result from newly discovered +hardware state. + +[Pre-ingestion](firmware-updates/pre-ingestion.md) is a related but separate +path. When automatic updates are enabled, it can update new hardware whose +firmware is below a configured minimum before NICo ingests and manages it. +After ingestion, drift detection keeps the hardware aligned with the configured +baseline. + +Not every firmware workflow begins with drift detection. Rack-scale compute +trays, NVSwitches, and power shelves can be updated through explicit API +requests. Some platforms require a vendor tool, a script, rack maintenance, or +a manually confirmed procedure. See [Choose the upgrade +path](#choose-the-upgrade-path) for the path used by each component. + +## How automatic work is scheduled + +Machine Update Manager schedules automatic firmware work for managed hosts and +DPUs. It does not install firmware itself; the managed-host state machine owns +the disruptive work and the return to service. + +On each scheduler pass, NICo: + +1. clears bookkeeping for updates that have completed; +2. counts the distinct hosts already in maintenance or an update workflow; +3. calculates the remaining capacity from the site-wide machine-update limit; +4. asks the enabled DPU and host firmware modules for eligible work; and +5. creates requests until the shared capacity is full. + +```mermaid +flowchart LR + A["Periodic scheduler pass"] --> B["Clear completed update markers"] + B --> C["Count hosts already updating"] + C --> D["Calculate shared capacity"] + D --> E["Apply DPU and host eligibility checks"] + E --> F["Create reprovisioning requests"] + F --> G["Managed-host state machine performs the work"] +``` + +Host and DPU work therefore compete for the same capacity, and a host with +several drifted DPUs counts once. Each module still applies its own state, +health, approval, and update-window checks before selecting a host. If the +calculated capacity is zero, drift remains pending and no automatic managed +machine update starts. + +Pre-ingestion firmware is not scheduled by Machine Update Manager. It has a +separate polling interval and concurrency limit because the hardware has not +yet entered the managed-host lifecycle. See the [Machine Update Manager +architecture](../architecture/overview.md#machine-update-manager) for the +broader ownership model. + +## Set the expected versions + +NICo does not discover the latest firmware version on its own. Before NICo can +detect drift, the site must define which firmware versions are expected. + +For host firmware, NICo uses a catalog organized by BMC vendor, host model, and +component. A component can define both: + +- `preingest_upgrade_when_below`: the threshold that triggers firmware work + before ingestion. +- `default`: the desired version used to detect drift after ingestion. + +For example, a host reporting BMC version `7.05.00.00` meets a configured +minimum of `7.00.00.00`, so ingestion can continue. If the default is +`7.10.30.00`, that host is still drifted after ingestion and becomes eligible +for the managed-host firmware workflow. + +Host definitions can come from the [Host Firmware Config +API](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/host-firmware-config/create-or-update-host-firmware-config), +legacy `metadata.toml` files, or static `host_models` configuration. NICo merges +these sources into one effective catalog. The API is the current configuration +method; `metadata.toml` is retained for existing deployments. + +DPU NIC firmware uses an accepted-version list in the site configuration +instead of the host catalog. Rack and component workflows receive their target +from an explicit request or their component backend. + +See [Configure firmware versions](firmware-updates/configuration.md) for the +configuration models, setting reference, examples, and source precedence. + +## Choose the upgrade path + +Use the table below to find the path for the hardware and component you are +updating. For hosts, the path also depends on whether ingestion has completed. + +| Your situation | Upgrade path | How it begins | |---|---|---| -| Host BMC, BIOS, UEFI, and platform firmware listed in host firmware metadata | Machine Update Manager and the managed-host state machine | Fleet host firmware drift and host firmware service windows. | -| DPU NIC firmware drift | Machine Update Manager and DPU reprovisioning | Automatic DPU reprovisioning when DPU NIC firmware is outside the configured baseline. | -| DPU BMC, UEFI, CEC, ERoT, and related DPU firmware | DPU reprovisioning | Verification and update as part of the DPU lifecycle. | -| NVIDIA platform procedures | Manual service procedure plus NICo completion gate | NVIDIA-managed platforms, such as GB200, that require a field procedure before NICo resumes automatic checks. | -| OEM platform procedures | OEM instructions plus NICo scheduling or script handoff | Vendor-specific packages, such as SMC/Supermicro platforms, where the OEM procedure is the authority for platform-specific steps. | -| Rack, compute tray, switch, and power shelf firmware | REST firmware APIs for rack and tray updates. Compute tray updates flow through Machine Update Manager and the managed-host lifecycle; switch and power shelf updates use component-manager. | Rack-level and component-level firmware updates. | - -Use `nico-admin-cli` for CLI examples. These commands talk to the NICo Core -gRPC API: - -```bash -nico-admin-cli -a -``` - -## Machine Update Manager - -Machine Update Manager is a scheduler. It does not flash firmware directly. On -each run it: - -1. Clears completed host and DPU update markers. -2. Counts machines already in maintenance or firmware update states. -3. Counts healthy and unhealthy hosts. -4. Computes how many additional updates can start. -5. Asks each enabled update module to start work until the site limit is - reached. - -Automated updates are selected only when the target is eligible. The normal -eligibility gates are: - -- The machine is known to NICo and is in a state that can be updated. -- The relevant update module is enabled. -- The machine needs firmware according to the configured baseline. -- The machine is not already in maintenance or another update flow. -- Site health and concurrency policy allow another host to leave service. -- A required update window is active when the firmware entry requires explicit - start. - -The actual firmware work runs inside the managed-host lifecycle. This keeps -host power, BMC operations, DPU reprovisioning, health reports, and state -transitions under one lifecycle controller. - -## Configuration - -Firmware behavior is controlled by site configuration and firmware metadata. - -### Machine Update Scheduling - -| Field | Default | Meaning | -|---|---:|---| -| `machine_update_run_interval` | `300s` when unset | How often Machine Update Manager runs. | -| `machine_updater.instance_autoreboot_period.start` | unset | Start of the UTC window when machines may automatically reboot for updates. | -| `machine_updater.instance_autoreboot_period.end` | unset | End of the UTC window when machines may automatically reboot for updates. | -| `machine_updater.max_concurrent_machine_updates_absolute` | unset | Hard cap on concurrent machine updates. | -| `machine_updater.max_concurrent_machine_updates_percent` | unset | Percentage cap on concurrent machine updates. When both caps are set, NICo uses the lower effective limit. | - -Example: - -```toml -machine_update_run_interval = 60 - -[machine_updater] -instance_autoreboot_period.start = "2026-05-22T01:00:00Z" -instance_autoreboot_period.end = "2026-05-22T05:00:00Z" -max_concurrent_machine_updates_absolute = 5 -max_concurrent_machine_updates_percent = 10 -``` - -### Host Firmware Settings - -| Field | Default | Meaning | -|---|---:|---| -| `firmware_global.autoupdate` | `false` | Enables automatic host firmware updates. | -| `firmware_global.host_enable_autoupdate` | `[]` | Host models or machine IDs to force-enable for host firmware autoupdate. | -| `firmware_global.host_disable_autoupdate` | `[]` | Host models or machine IDs to force-disable for host firmware autoupdate. | -| `firmware_global.run_interval` | `30s` | Firmware manager polling interval. | -| `firmware_global.max_uploads` | `4` | Maximum concurrent firmware uploads. | -| `firmware_global.concurrency_limit` | `16` | Maximum concurrent firmware flashing operations. | -| `firmware_global.firmware_directory` | `/opt/nico/firmware` | Directory for firmware binaries and `metadata.toml` files. | -| `firmware_global.host_firmware_upgrade_retry_interval` | `60m` | Delay before retrying a failed host firmware upgrade. | -| `firmware_global.instance_updates_manual_tagging` | `true` | Requires manual tagging before host firmware updates are applied. | -| `firmware_global.no_reset_retries` | `false` | Disables retry logic after BMC resets during firmware operations. | -| `firmware_global.hgx_bmc_gpu_reboot_delay` | `30s` | Delay after GPU reboot before the HGX BMC can be accessed. | -| `firmware_global.requires_manual_upgrade` | `false` | Forces firmware upgrades through a manual completion gate. | -| `firmware_global.max_concurrent_bfb_copies` | `10` | Maximum concurrent BFB copies. | - -Example: - -```toml -[firmware_global] -autoupdate = true -host_enable_autoupdate = ["PowerEdge R750"] -host_disable_autoupdate = [] -run_interval = "30s" -max_uploads = 4 -concurrency_limit = 16 -firmware_directory = "/opt/nico/firmware" -host_firmware_upgrade_retry_interval = "60m" -requires_manual_upgrade = false -``` - -### DPU Firmware Settings - -| Field | Default | Meaning | -|---|---:|---| -| `dpu_config.dpu_nic_firmware_initial_update_enabled` | `false` | Enables DPU NIC firmware updates on initial discovery. | -| `dpu_config.dpu_nic_firmware_reprovision_update_enabled` | `true` | Enables DPU NIC firmware updates during reprovisioning. | -| `dpu_config.dpu_models` | BF2/BF3 defaults | DPU firmware definitions. | -| `dpu_config.dpu_nic_firmware_update_versions` | BF2/BF3 NIC versions | Accepted DPU NIC firmware versions. Firmware drift outside this set can trigger automated DPU reprovisioning. | -| `dpu_config.dpu_enable_secure_boot` | `false` | Enables the secure boot flow for DPU provisioning through Redfish. | - -## Firmware Baselines - -NICo compares observed firmware inventory with configured firmware baselines. -Host baselines are loaded from embedded configuration and from -`metadata.toml` files under `firmware_global.firmware_directory`. Metadata can -define vendor and model matching, component ordering, known firmware versions, -default versions, minimum pre-ingestion versions, and whether explicit start is -required. - -List host firmware entries known to NICo: - -```bash -nico-admin-cli -a firmware show -``` - -The output includes vendor, model, component type, inventory-name match, -version, and whether the update needs explicit start. - -Keep live "latest firmware version" tables in a single baseline source and link -to that source from site runbooks. Do not copy live version tables into this -operations page; version tables change independently from the update workflow. -If a site mirrors firmware versions into a wiki or dashboard, make the mirror -point back to the same source-controlled baseline or approved version catalog. - -To update a host firmware baseline: - -1. Add or update the firmware metadata for the target vendor and model. -2. Place the firmware binary where `firmware_global.firmware_directory` points, - or provide the approved URL/script metadata used by the site. -3. Mark the intended version as the default for the component. -4. Verify the new baseline with `firmware show`. -5. Apply the update through a service window or machine-specific update window. - -Example host firmware metadata: - -```toml -[host_models.dell_r750] -vendor = "Dell" -model = "PowerEdge R750" -ordering = ["uefi", "bmc"] - -[host_models.dell_r750.components.bmc] -current_version_reported_as = "^Installed-.*__iDRAC." -preingest_upgrade_when_below = "0.5" -known_firmware = [ - { version = "1.1", filename = "/opt/nico/firmware/dell/r750_bmc_1.1.fw", checksum = "", default = true }, -] - -[host_models.dell_r750.components.uefi] -current_version_reported_as = "^Installed-.*__BIOS.Setup." -preingest_upgrade_when_below = "0.5" -known_firmware = [ - { version = "2.0", filename = "/opt/nico/firmware/dell/r750_uefi_2.0.fw", checksum = "", default = true }, -] -``` - -## Host Firmware Updates - -Host firmware updates use Redfish inventory and firmware metadata. Common host -components include BMC, BIOS, UEFI, HGX BMC, GPU, NIC, and platform-specific -firmware, depending on the platform metadata. - -During a host firmware update, NICo can: - -- Upload firmware through Redfish or run an approved firmware script. -- Poll Redfish tasks and firmware inventory. -- Reset the BMC or host when required by the platform. -- Re-check firmware versions after activation. -- Apply a `HostUpdateInProgress` health report that prevents allocation while - update work is active. - -Enable, disable, or clear machine-specific auto-update behavior: - -```bash -nico-admin-cli -a machine auto-update --machine --enable -nico-admin-cli -a machine auto-update --machine --disable -nico-admin-cli -a machine auto-update --machine --clear -``` - -Set an explicit firmware update window for one or more machines: - -```bash -nico-admin-cli -a managed-host start-updates \ - --machines \ - --start 2026-05-22T01:00:00-0700 \ - --end 2026-05-22T05:00:00-0700 -``` - -Cancel pending start windows: - -```bash -nico-admin-cli -a managed-host start-updates \ - --machines \ - --cancel -``` - -Request host reprovisioning when the host must be put back through the -managed-host firmware path: - -```bash -nico-admin-cli -a host reprovision set \ - --id \ - --update-firmware \ - --update-message "" -``` - -## DPU Firmware Updates - -DPU firmware is managed as part of the managed-host lifecycle. NICo tracks: - -| Component | Inventory name | Meaning | -|---|---|---| -| DPU NIC firmware | `DPU_NIC` | Primary NIC firmware on the BlueField. | -| DPU BMC firmware | `BMC_Firmware` | DPU management controller firmware. | -| DPU UEFI firmware | `DPU_UEFI` | DPU boot firmware. | -| ATF / ERoT firmware | `Bluefield_FW_ERoT` | Arm Trusted Firmware or External Root of Trust. | - -Machine Update Manager uses DPU NIC firmware drift as the automatic trigger. -During DPU reprovisioning, NICo verifies and updates the DPU firmware set -against the configured baseline. - -Inspect DPU firmware status: - -```bash -nico-admin-cli -a dpu versions -``` - -For the full DPU firmware flow, see -[DPU Lifecycle Management](../dpu-management/dpu-lifecycle-management.md#firmware-upgrades). - -## Manual Platform Updates - -Some platforms require a manual field procedure before NICo can complete the -firmware workflow. - -### NVIDIA Platforms - -For NVIDIA-managed platforms, follow the approved NVIDIA service procedure for -the exact platform and firmware package. GB200 is the common example for this -flow. NICo normally runs the automated firmware lifecycle. Use the manual -procedure only when the site runbook or platform support path requires a manual -GB200 update gate. - -A GB200 firmware service procedure can cover several component groups: - -| Component group | Purpose | -|---|---| -| NVSwitch trays | Updates switch firmware before the rack is returned to service. | -| GB200 compute tray BMC | Updates the compute tray management controller used for out-of-band control. | -| HGX firmware | Updates the GPU/HGX firmware package for the compute tray. | -| System BIOS or UEFI | Updates host boot firmware and platform initialization behavior. | -| Auxiliary power or BMC power state | Applies platform power-state changes required for firmware activation. | -| CPU node BMC and BIOS | Updates CPU node management and boot firmware through the applicable OEM procedure. | -| TPM state | Clears or resets TPM state when required by the platform procedure. | -| NVMe SSD firmware and secure erase | Updates storage firmware and returns storage to the expected clean state. | -| CX7 firmware | Updates ConnectX firmware used by the host networking stack. | -| BlueField-3 firmware | Updates DPU NIC and boot-related firmware where the GB200 rack uses BF3 DPUs. | -| Power cycle and validation | Activates pending firmware and verifies platform health, including NVLink status. | - -The exact order, package names, credentials, BMC addresses, and validation -commands are site- and release-specific. Keep those details in the approved -GB200 service procedure. This Operations guide documents where NICo waits and -how to resume the lifecycle after that procedure is complete. - -When manual firmware upgrade is required, NICo moves the managed host to a -manual waiting state. Complete the approved GB200 firmware procedure first. -After the field procedure is complete, mark the manual firmware upgrade -complete so NICo can resume automatic checks: - -```bash -nico-admin-cli -a host reprovision mark-manual-upgrade-complete --id -``` - -### OEM Platforms - -For OEM platforms, consult the OEM support site for the exact platform and -firmware package before starting work. The OEM support site is the final -authority for vendor-specific prerequisites, package selection, activation -steps, and recovery. - -SMC/Supermicro is one example of this path. NICo can use the Redfish firmware -workflow when firmware metadata identifies the target component and package. -The Supermicro Redfish implementation supports: - -| Component or action | NICo behavior | -|---|---| -| Lockdown state | Reads and updates the Supermicro `SysLockdown` setting before firmware work when required. | -| BMC firmware | Uses Redfish multipart upload with the BMC target and Supermicro OEM preserve/backup settings. | -| BIOS or UEFI firmware | Uses Redfish multipart upload with the BIOS target and Supermicro OEM preserve settings. | -| CPLD firmware | Supports Supermicro motherboard and backplane CPLD Redfish inventory targets. | - -NICo can schedule, track, or hand off the update when the site has integrated -an approved firmware package or script. Use the OEM procedure to confirm the -exact package, update order, activation requirements, and recovery steps. - -## Rack and Component Firmware - -Use the NVIDIA Forge REST API for rack and tray firmware updates. The API -accepts a site ID and optional target firmware version, starts the update -workflow, and returns task IDs for tracking. - -REST API entry points: - -| Scope | Method and path | Use | -|---|---|---| -| Single rack | `PATCH /v2/org/{org}/nico/rack/{id}/firmware` | Update firmware for one rack. | -| Rack batch | `PATCH /v2/org/{org}/nico/rack/firmware` | Update firmware for racks selected by filter, or all racks in the site when no filter is supplied. | -| Single tray | `PATCH /v2/org/{org}/nico/tray/{id}/firmware` | Update firmware for one compute tray. | -| Tray batch | `PATCH /v2/org/{org}/nico/tray/firmware` | Update firmware for trays selected by filter, or all trays in the site when no filter is supplied. | - -Request fields: - -| Field | Required | Use | -|---|---|---| -| `siteId` | Yes | Site that owns the rack or tray targets. | -| `version` | No | Target firmware version. Omit only when the site workflow selects the target version. | -| `filter` | Batch only | Selects the batch targets. Rack batch filters by rack name. Tray batch filters by rack ID, rack name, tray ID, component ID, or tray type. | - -Batch tray filters follow these constraints: - -- Use either `rackId` or `rackName`, not both. -- Do not combine a rack filter with `ids` or `componentIds`. -- Use `type` when filtering by `componentIds`. - -The API requires provider-admin authorization and returns task IDs for tracking. - -Single-rack example: - -```bash -curl -X PATCH "https:///v2/org//nico/rack//firmware" \ - -H "Authorization: Bearer " \ - -H "Content-Type: application/json" \ - -d '{ - "siteId": "", - "version": "" - }' -``` - -Batch tray example for a rack: - -```bash -curl -X PATCH "https:///v2/org//nico/tray/firmware" \ - -H "Authorization: Bearer " \ - -H "Content-Type: application/json" \ - -d '{ - "siteId": "", - "version": "", - "filter": { - "rackName": "" - } - }' -``` - -The response contains task IDs: - -```json -{ - "taskIds": [ - "" - ] -} -``` - -Use component-manager only for lower-level switch and power shelf firmware -workflows when a REST workflow is not available for that component: - -```bash -nico-admin-cli -a component-manager update-firmware switch \ - --switch-id \ - --target-version \ - --component bmc,cpld,bios,nvos - -nico-admin-cli -a component-manager update-firmware power-shelf \ - --power-shelf-id \ - --target-version \ - --component pmc,psu -``` - -Check component firmware status: - -```bash -nico-admin-cli -a component-manager get-firmware-update-status switch --switch-id -nico-admin-cli -a component-manager get-firmware-update-status power-shelf --power-shelf-id -``` - -## Monitor Progress - -Start with the object being updated, then move to logs and metrics when progress -is unclear. - -| Task | Command | -|---|---| -| Show managed-host state and handler outcome | `nico-admin-cli -a managed-host show ` | -| Show machine details and health reports | `nico-admin-cli -a machine show ` | -| Show host firmware baseline entries | `nico-admin-cli -a firmware show` | -| Show DPU firmware status | `nico-admin-cli -a dpu versions` | -| Start rack or tray firmware update workflow | Use the rack and tray REST firmware endpoints. The response returns `taskIds`. | -| Show switch or power shelf component firmware status | `nico-admin-cli -a component-manager get-firmware-update-status ...` | - -Useful metrics: - -| Metric | Meaning | -|---|---| -| `carbide_pending_host_firmware_update_count` | Host machines that need a host firmware update. | -| `carbide_active_host_firmware_update_count` | Host machines actively updating firmware. | -| `carbide_pending_dpu_nic_firmware_update_count` | Machines with DPU NIC firmware drift. | -| `carbide_unavailable_dpu_nic_firmware_update_count` | Machines with DPU NIC firmware drift that are not currently available for update. | -| `carbide_running_dpu_updates_count` | Machines running DPU firmware update work. | -| `carbide_dpu_firmware_version_count` | DPUs reporting a firmware version. | -| `carbide_preingestion_waiting_download` | Pre-ingestion hosts waiting for firmware downloads on other machines to complete. | -| `carbide_preingestion_waiting_installation` | Pre-ingestion hosts with firmware uploaded and installation in progress. | - -## Recovery - -If a firmware update does not progress: - -1. Inspect the managed-host state, REST task IDs, or component status for the - failing object. -2. Check the health reports and handler outcome for the reason NICo is waiting. -3. Check `nico-api` logs for Redfish task failures, BMC reachability errors, - script failures, REST firmware task failures, RMS backend errors, or - component-manager job errors. -4. Follow the platform or OEM recovery procedure before retrying a failed - firmware operation. -5. Retry through the same NICo workflow after the underlying platform condition - is corrected. - -A host firmware failure can place the host in a failed firmware-upgrade state. -NICo retries according to `firmware_global.host_firmware_upgrade_retry_interval` -where retry is supported. A rack or component firmware failure should be -tracked through the REST task IDs returned by the rack or tray firmware API, -or through component-manager status for switch and power shelf targets. - -After the platform condition is corrected, reset a failed host reprovisioning -flow only when the site runbook calls for it: - -```bash -nico-admin-cli -a managed-host reset-host-reprovisioning --machine -``` +| A host has not completed ingestion and a reported component version is below `preingest_upgrade_when_below`. | [Pre-ingestion](firmware-updates/pre-ingestion.md) | The pre-ingestion manager detects the condition. When global automatic updates are enabled, it selects the default or pre-ingestion-only firmware entry and starts the update. | +| Firmware on a managed host differs from the effective default in the merged firmware catalog. | [Host firmware](firmware-updates/host-firmware.md) | When host automatic updates are enabled, NICo detects the drift and schedules host reprovisioning. If `explicit_start_needed` is enabled, NICo first waits for the host's update window. | +| A managed DPU NIC reports a version outside `dpu_nic_firmware_update_versions`. | [DPU NIC firmware](firmware-updates/dpu-firmware.md#dpu-nic-firmware) | When DPU NIC updates are enabled, NICo detects the drift and schedules DPU reprovisioning. | +| A DPU component other than the NIC needs an update. | [DPU firmware components](firmware-updates/dpu-firmware.md#components-covered) | Request DPU reprovisioning explicitly. Applying the BFB updates the bundled firmware and verifies selected components; non-NIC drift is not an independent automatic trigger. | +| A rack, compute tray, NVSwitch, or power shelf needs an update. | [Rack and component firmware](firmware-updates/rack-component-firmware.md) | An operator requests the update through the appropriate API. NICo routes it to the controller or backend responsible for that component. | diff --git a/docs/operations/firmware-updates/configuration.md b/docs/operations/firmware-updates/configuration.md new file mode 100644 index 0000000000..6ecf762126 --- /dev/null +++ b/docs/operations/firmware-updates/configuration.md @@ -0,0 +1,382 @@ +# Configure firmware versions + +NICo needs a site-specific definition of acceptable firmware before it can +detect drift or choose an upgrade target. Host firmware and DPU NIC firmware +use different configuration models. Rack and component updates use a target +supplied by the request or selected by their backend. + +The firmware catalog answers **what firmware NICo recognizes and should +install**. Start there: define host firmware through the [Host Firmware Config +API](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/host-firmware-config/create-or-update-host-firmware-config) +or legacy metadata. Later sections cover DPU baselines and the site controls +that determine **when NICo looks for work** and **how much work it may start**. + +## Host firmware catalog + +Each host firmware definition is associated with a BMC vendor and host model. +The definition can come from the Host Firmware Config API, a legacy +`metadata.toml` file, or the static `host_models` configuration. NICo merges +these sources into one effective catalog. + +During site exploration, Redfish reports the BMC vendor and host model. NICo +uses that pair to select the corresponding definition from the catalog. A +definition can contain several components, such as BMC, UEFI, CPLD, HGX BMC, +GPU, NIC, or CX7 firmware. + +A definition contains model-level settings and one or more component entries: + +| Setting | Scope | Meaning | +|---|---|---| +| `vendor` | Model | BMC vendor to which the definition applies. | +| `model` | Model | Host model to which the definition applies. Together, `vendor` and `model` identify the definition. | +| `components` | Model | Groups the component definitions, keyed by component type. | +| `ordering` | Model | Lists the components in the order NICo should update them. | +| `explicit_start_needed` | Model | When enabled, drift alone does not start the update. | +| `type` | Component | Identifies the component as BMC, UEFI, CEC, NIC, CPLD, HGX BMC, GPU, CX7, or a combined BMC and UEFI package. In legacy TOML, the type is the key below `components`. | +| `current_version_reported_as` | Component | Regular expression used to find the component in the firmware inventory reported by the BMC. The API derives this matcher from the vendor, model, and component type. | +| `preingest_upgrade_when_below` | Component | Sets the threshold for pre-ingestion firmware work. A lower reported version triggers an update when global automatic updates are enabled. | +| `known_firmware` | Component | Lists the versions NICo can install and the artifacts used to install them. | +| `version` | Firmware version | Version string NICo expects the firmware inventory to report after installation. | +| `default` | Firmware version | Marks the desired version used for drift detection after ingestion. | +| `preingestion_exclusive_config` | Firmware version | Marks a version for use during pre-ingestion without making it the steady-state default. | +| `files` | Firmware version | Structured list of firmware artifacts. Each entry provides a filename or URL and can include a SHA-256 digest. The API exposes this list as `artifacts`. | +| `install_only_specified` | Firmware version | Tells Redfish to install only the component identified by the firmware definition when the package contains multiple components. | +| `power_drains_needed` | Firmware version | Number of full power-drain cycles required to activate the firmware after installation. | +| `pre_update_resets` | Firmware version | Requests the platform-specific reset sequence before NICo starts the installation. | +| `script` | Firmware version | Legacy path that hands the update to a local script instead of using the normal Redfish upload path. | + +`current_version_reported_as` is matched against firmware inventory entry +identifiers, not against version strings. For example, +`^Installed-.*__iDRAC.` selects the iDRAC entry from a Dell Redfish firmware +inventory. NICo then reads the installed version from that matching entry. + +Legacy metadata also accepts the older artifact fields `filename`, `filenames`, +`url`, and `checksum`; new API entries use URL-based `artifacts`. The data model +still parses `mandatory_upgrade_from_priority` and `scout`, but the current +upgrade workflow does not use them as firmware-policy controls, so they are not +listed as active settings here. + +### Minimum and default versions + +The pre-ingestion minimum is an upgrade trigger, not the upgrade target. When a +host is below the minimum and global automatic updates are enabled, NICo +normally installs the version marked as the default. A firmware version can +instead set `preingestion_exclusive_config = true` when the platform needs a +different target during ingestion. + +For example, assume the BMC minimum is `7.00.00.00` and the default is +`7.10.30.00`: + +| Reported version | Result | +|---|---| +| `6.50.00.00` | Below the minimum. With automatic updates enabled, NICo updates the BMC during pre-ingestion, using `7.10.30.00` as the target. | +| `7.05.00.00` | Meets the minimum, so ingestion can continue. After ingestion it is still drifted from `7.10.30.00`. | +| `7.10.30.00` | Meets the minimum and matches the desired version. No firmware update is needed. | + +See [Pre-ingestion firmware](pre-ingestion.md) for minimum-version enforcement +and [managed host firmware](host-firmware.md) for post-ingestion drift handling. + +### Configure host firmware through the API + +The current way to configure host firmware is the [Create or Update Host +Firmware Config](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/host-firmware-config/create-or-update-host-firmware-config) +operation: + +```text +PUT /v2/org/{org}/nico/firmware-config/host +``` + +The request is site-scoped and keyed by `(vendor, model)`. NICo stores the +configuration in `host_firmware_config` and uses it on subsequent discovery, +pre-ingestion, drift detection, and upgrade passes. This allows a site to +change its firmware catalog without changing the NICo deployment or placing +metadata files on the Core filesystem. + +The endpoint requires Provider Admin authorization. The following request +defines one BMC version for a Dell PowerEdge R760: + +```bash +curl -X PUT "https:///v2/org//nico/firmware-config/host" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "siteId": "", + "vendor": "Dell", + "model": "PowerEdge R760", + "ordering": ["BMC"], + "components": [ + { + "type": "BMC", + "preingestUpgradeWhenBelow": "7.00.00.00", + "firmware": [ + { + "version": "7.10.30.00", + "default": true, + "artifacts": [ + { "url": "https:///" } + ] + } + ] + } + ] + }' +``` + +The request does not contain an inventory-matching regular expression. Core +derives that expression from its supported vendor, model, and component +mappings. The request is rejected when no mapping exists. + +`PUT` is an upsert, not a complete replacement: + +- The first request for a vendor and model must provide an ordering that + contains every configured component. +- Each component must have exactly one default firmware version after the + request is merged. +- Later requests can add components or firmware versions. Versions are merged + by version string, and omitted components remain unchanged. +- Marking a new version as the default clears the previous default for that + component. + +The [Delete Host Firmware +Config](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/host-firmware-config/delete-host-firmware-config) +operation removes the runtime entry. If the same vendor and model also exists +in static or legacy metadata, that lower-priority definition becomes effective +again. + +### Legacy `metadata.toml` configuration + +Before the Host Firmware Config API was introduced, host firmware definitions +were deployed as `metadata.toml` files. This is the legacy configuration path. +It remains supported for existing deployments, but new host firmware +definitions should use the API. + +`firmware_global.firmware_directory` points to a directory containing one +subdirectory per metadata entry. NICo looks for a file named `metadata.toml` in +each of those subdirectories. For example: + +```text +/opt/nico/firmware/ +`-- dell-r760-bmc-7.10.30/ + `-- metadata.toml +``` + +Each file describes one vendor and model and can define one or more firmware +components. The equivalent legacy definition for the API example above is: + +```toml +vendor = "Dell" +model = "PowerEdge R760" +ordering = ["bmc"] + +[components.bmc] +current_version_reported_as = "^Installed-.*__iDRAC." +preingest_upgrade_when_below = "7.00.00.00" + +[[components.bmc.known_firmware]] +version = "7.10.30.00" +url = "https:///" +default = true +``` + +NICo also supports host definitions embedded in the static `host_models` +configuration. This is the base catalog supplied with the NICo deployment and +uses the same firmware data model as `metadata.toml`. + +NICo builds the effective host firmware catalog in this order: + +| Applied | Source | +|---|---| +| First | Static `host_models` configuration. | +| Next | Legacy `metadata.toml` files under `firmware_global.firmware_directory`. | +| Last (highest precedence) | Runtime `host_firmware_config` entries created through the API. | + +Higher-priority sources overlay matching vendor, model, and component entries; +they do not discard unrelated lower-priority entries. For example, if a +`metadata.toml` file defines BMC `7.00.00.00` as the default and an API request +adds BMC `7.10.30.00` as the default, both remain known versions, but +`7.10.30.00` becomes the effective target. + +Use the CLI to inspect the effective default versions after all three sources +have been merged: + +```bash +nico-admin-cli -a firmware show +``` + +## DPU firmware + +DPU firmware does not use the Host Firmware Config API. Its site configuration +has two related parts: an accepted-version list that triggers automatic work, +and a model catalog used to verify the result. + +| Setting | Default | Meaning | +|---|---|---| +| `dpu_config.dpu_nic_firmware_initial_update_enabled` | `false` | Compatibility setting for initial DPU setup. The current state machine records it but does not use it to suppress BFB installation or verification. | +| `dpu_config.dpu_nic_firmware_reprovision_update_enabled` | `true` | Enables automatic DPU NIC drift selection in Machine Update Manager. Explicit DPU reprovisioning remains available when disabled. | +| `dpu_config.dpu_nic_firmware_update_versions` | Built-in BF2 and BF3 NIC versions | Accepted NIC versions used by the classic automatic drift detector. | +| `dpu_config.dpu_models` | Built-in BF2 and BF3 definitions | BMC, ERoT/CEC, NIC, and UEFI baselines associated with each DPU model. | +| `dpu_config.dpu_enable_secure_boot` | `false` | Allows classic reprovisioning to install the BFB through Redfish and enable secure boot when every attached DPU supports that path. Otherwise NICo uses UEFI HTTP network installation. | + +### Automatic DPU NIC trigger + +NICo checks the reported DPU NIC version against +`dpu_config.dpu_nic_firmware_update_versions`: + +```toml +[dpu_config] +dpu_nic_firmware_update_versions = ["32.43.1014", "32.44.1030"] +``` + +This is an accepted-version list rather than a single default. A DPU reporting +either version in the example is considered compliant. A version outside the +list is drifted and, when DPU NIC updates are enabled, makes the DPU eligible +for reprovisioning. Allowing more than one version is useful during a staged +rollout where both the old and new versions must temporarily remain valid. + +`dpu_nic_firmware_reprovision_update_enabled` enables this automatic drift +module. It does not prevent an operator from requesting DPU reprovisioning. + +### Post-reprovision baseline + +The `dpu_models` catalog describes the component versions expected after a +classic DPU reprovision. Entries are keyed by NICo's normalized DPU model, +currently `bluefield2` or `bluefield3`, and use the same firmware structures as +the static host catalog: + +```toml +[dpu_config.dpu_models.bluefield3] +vendor = "Nvidia" +model = "Bluefield 3 SmartNIC Main Card" +ordering = ["bmc", "cec"] + +[dpu_config.dpu_models.bluefield3.components.nic] +current_version_reported_as = "DPU_NIC" + +[[dpu_config.dpu_models.bluefield3.components.nic.known_firmware]] +version = "32.44.1030" +default = true +``` + +The built-in catalog defines BMC, CEC/ERoT, NIC, and UEFI entries for each +model. The current reprovisioning verifier compares BMC, CEC/ERoT, and NIC. It +selects the last non-pre-ingestion-only known version for each component; in +normal configurations, keep one steady-state entry for each component. + +NICo applies these versions by installing or booting the deployed BlueField +Bundle, not by uploading the `known_firmware` entries as independent packages. +The accepted NIC list, DPU model catalog, and firmware carried by `forge.bfb` +must therefore be updated together. For example, if the BFB installs NIC +version `32.44.1030`, that version must be accepted by +`dpu_nic_firmware_update_versions` and should be the NIC baseline in the +matching `dpu_models` entry. + +See [DPU firmware upgrades](dpu-firmware.md) for how NICo acts on this +configuration. + +## Operational controls + +The settings in this section are fields in NICo Core's site configuration. They +control the automatic paths; they do not define component target versions. + +### Managed-machine scheduling + +Machine Update Manager uses one capacity budget for managed-host and DPU +firmware work: + +| Setting | Default | Meaning | +|---|---|---| +| `machine_update_run_interval` | `300` | Seconds between Machine Update Manager passes. | +| `machine_updater.max_concurrent_machine_updates_absolute` | unset | Maximum number of distinct hosts that may be in machine-update workflows. Set this to a positive value to allow automatic managed-host or DPU work. | +| `machine_updater.max_concurrent_machine_updates_percent` | unset | Intended percentage-based capacity. The current implementation calculates this incorrectly; leave it unset and use the absolute limit. | +| `machine_updater.instance_autoreboot_period.start` | unset | Start of a one-time UTC window in which an assigned host may consume an existing update request without tenant approval. | +| `machine_updater.instance_autoreboot_period.end` | unset | End of that one-time UTC window. | + +For example, this allows at most five managed hosts to be in update workflows +and gives assigned hosts a four-hour automatic reboot window: + +```toml +machine_update_run_interval = 300 + +[machine_updater] +max_concurrent_machine_updates_absolute = 5 +instance_autoreboot_period.start = "2026-08-15T01:00:00Z" +instance_autoreboot_period.end = "2026-08-15T05:00:00Z" +``` + +The autoreboot period is not the per-host firmware window used by models with +`explicit_start_needed`. Operators open that window through the managed-host +API. See [Scheduling gates](host-firmware.md#scheduling-gates). + +### Firmware execution + +`firmware_global` controls host firmware execution and the separate +pre-ingestion manager: + +| Setting | Default | Meaning | +|---|---|---| +| `firmware_global.autoupdate` | `false` | Enables pre-ingestion upgrades and supplies the default automatic-update policy for managed hosts. A per-machine policy can override the managed-host default. | +| `firmware_global.run_interval` | `30s` | Interval between pre-ingestion manager passes. | +| `firmware_global.max_uploads` | `4` | Shared limit for concurrent host firmware uploads. | +| `firmware_global.concurrency_limit` | `16` | Maximum number of pre-ingestion endpoints processed concurrently. This is separate from Machine Update Manager capacity. | +| `firmware_global.firmware_directory` | `/opt/nico/firmware`, otherwise `/opt/carbide/firmware` | Directory containing legacy metadata and local firmware artifacts. | +| `firmware_global.firmware_download_cache_directory` | `/mnt/persistence/fw/download-cache` | Writable cache for downloaded firmware artifacts. | +| `firmware_global.host_firmware_upgrade_retry_interval` | `60m` | Delay before the managed-host state machine retries a failed host firmware upgrade. | +| `firmware_global.instance_updates_manual_tagging` | See below | When `true`, automatic host selection is limited to unassigned hosts in top-level `Ready`. When `false`, assigned hosts can also receive a pending request. | +| `firmware_global.no_reset_retries` | `false` | Disables the normal retry handling for BMC reset failures during pre-ingestion and managed-host firmware work. | +| `firmware_global.hgx_bmc_gpu_reboot_delay` | `30s` | Delay after a GPU reboot before NICo accesses the HGX BMC again. | +| `firmware_global.requires_manual_upgrade` | `false` | Adds a manually confirmed upgrade gate for MNNVL-capable hosts. | +| `firmware_global.max_concurrent_bfb_copies` | `10` | Maximum concurrent BFB copies performed by the pre-ingestion manager. | + +Set `instance_updates_manual_tagging` explicitly. When a present +`firmware_global` table omits the field, deserialization uses `true`; when the +entire table is omitted, the current whole-structure default uses `false`. + +A staging configuration can make the policy explicit while leaving site-wide +automatic selection disabled: + +```toml +[firmware_global] +autoupdate = false +run_interval = "30s" +max_uploads = 4 +concurrency_limit = 16 +host_firmware_upgrade_retry_interval = "60m" +instance_updates_manual_tagging = true +requires_manual_upgrade = false +max_concurrent_bfb_copies = 10 +``` + +The legacy `host_enable_autoupdate` and `host_disable_autoupdate` fields do not +provide a reliable model allowlist. Use the [per-machine automatic-update +policy](host-firmware.md#enable-or-disable-one-host) instead. + +## Roll out a new baseline + +Treat a baseline change as a site-wide rollout, even when the first update is a +canary: + +1. Update the canonical configuration source. Do not maintain a second live + version table in this guide or a site runbook. +2. Make every referenced host artifact available. For DPU firmware, update the + BFB, accepted NIC versions, and `dpu_models` together. +3. Check host component ordering, defaults, pre-ingestion thresholds, and + inventory matchers. Then inspect the merged host catalog with `firmware + show`. +4. Configure a nonzero shared capacity, explicit-start windows, and assigned-host + approval policy. Keep site-wide automatic selection disabled while validating + one host through a per-machine host policy or an explicit DPU reprovisioning + request. +5. After verification, enable the intended site-wide automatic policy. +6. Monitor the applicable workflow until inventory reports the configured + versions and the host has returned to service. + +Changing a pre-ingestion threshold while `firmware_global.autoupdate` is enabled +can affect every matching endpoint on the next pre-ingestion pass. Validate the +catalog and artifact availability before applying that change. + +## Rack and component firmware + +Rack and component firmware do not use either of the configuration models +above. Their target version comes from the update request or from the default +selected by the rack or component backend. See [Rack and component firmware +upgrades](rack-component-firmware.md). diff --git a/docs/operations/firmware-updates/dpu-firmware.md b/docs/operations/firmware-updates/dpu-firmware.md new file mode 100644 index 0000000000..9c63981ed3 --- /dev/null +++ b/docs/operations/firmware-updates/dpu-firmware.md @@ -0,0 +1,248 @@ +# DPU firmware upgrades + +DPU firmware is updated as part of DPU reprovisioning. NICo does not upload a +separate package for each DPU component. Instead, it installs or boots the +site's BlueField Bundle (BFB), power-cycles the platform, and then checks the +firmware reported by the DPU BMC. + +Keep two kinds of configuration separate: + +- `dpu_nic_firmware_update_versions` determines whether DPU NIC drift should + automatically start reprovisioning. +- `dpu_models` contains the BMC, ERoT/CEC, and NIC versions that NICo expects + after the BFB has been applied. + +These values and the BFB must describe the same release. See [Configure +firmware versions](configuration.md#dpu-firmware) before enabling automatic +updates. + +## Components covered + +The classic, non-DPF reprovisioning path uses the following Redfish inventory: + +| Component | Redfish inventory match | Role in the workflow | +|---|---|---| +| DPU OS and bundled firmware | `DPU_OS` / BFB | NICo installs `forge.bfb` through Redfish when supported, or boots the DPU into the network-install path. The BFB carries the DPU software and associated firmware payloads. | +| DPU NIC | `DPU_NIC` | Its reported version is the automatic drift trigger. NICo also verifies it after reprovisioning. | +| DPU BMC | An inventory ID containing `BMC_Firmware` | Verified after reprovisioning against the DPU model catalog. | +| ERoT/CEC | `Bluefield_FW_ERoT` | Verified after reprovisioning against the DPU model catalog. | +| DPU UEFI | `DPU_UEFI` | Read for inventory and topology data, but not compared in the current post-reprovision verification loop. | + +The distinction matters: BMC or ERoT/CEC drift alone does not make Machine +Update Manager select a DPU. A NIC mismatch starts the workflow; applying the +BFB is expected to bring the other verified components to the corresponding +baseline. + +## DPU NIC firmware + +### Drift detection + +Machine Update Manager periodically reads the DPU NIC version discovered for +each managed DPU. It trims surrounding whitespace and checks for an exact +match in `dpu_config.dpu_nic_firmware_update_versions`. + +For example: + +```toml +[dpu_config] +dpu_nic_firmware_reprovision_update_enabled = true +dpu_nic_firmware_update_versions = ["32.43.1014", "32.44.1030"] +``` + +A DPU reporting either listed version is compliant. A DPU reporting +`32.42.1000` is drifted. The list does not identify which version NICo will +install; the deployed BFB determines the resulting firmware versions. + +Setting `dpu_nic_firmware_reprovision_update_enabled = false` removes the DPU +NIC module from Machine Update Manager. NICo then stops detecting and +scheduling classic DPU NIC drift automatically. It does not disable DPU +reprovisioning requested through the API. + +### Eligibility and scheduling + +Drift is evaluated per DPU, but scheduling and disruption are managed per +host. If several DPUs attached to one host are drifted, NICo groups them into +one host update and counts that host once against the shared machine-update +limit. + +NICo automatically selects a host only when all of the following are true: + +| Check | Requirement | +|---|---| +| Managed-host state | The host is in top-level `Ready`. An assigned host is not automatically selected. | +| Health | The aggregate host health report has no alerts. | +| Existing work | The host is not already in another machine-update workflow, and none of its DPUs already has a reprovisioning request. | +| Site capacity | The shared `machine_updater.max_concurrent_machine_updates_*` policy has room for another host. DPU and host firmware work use the same budget. | + +When the host is eligible, Machine Update Manager creates a reprovisioning +request on each drifted DPU. The managed-host controller adds a +`HostUpdateInProgress` health alert with `PreventAllocations`, restarts the +selected DPUs, and enters DPU reprovisioning. + +```mermaid +flowchart TD + A["Read discovered DPU NIC version"] --> B{"Version is accepted?"} + B -- Yes --> C["No action"] + B -- No --> D{"Host is healthy, Ready, and within the update budget?"} + D -- No --> E["Report as unavailable and retry later"] + D -- Yes --> F["Create DPU reprovisioning request"] + F --> G["Mark host unavailable for allocation"] + G --> H["Reprovision DPU and verify firmware"] +``` + +### Reprovisioning and verification + +The installation mechanism depends on how the DPU is managed and on platform +capabilities: + +```mermaid +flowchart TD + A["DPU reprovisioning starts"] --> B{"DPF-managed DPU?"} + B -- Yes --> C["Ask DPF to reprovision the DPU"] + B -- No --> D{"Secure boot enabled and all DPUs support Redfish BFB install?"} + D -- Yes --> E["Install forge.bfb through Redfish UpdateService"] + D -- No --> F["Boot the DPU into the UEFI HTTP network-install path"] + C --> G["Wait for DPU readiness"] + E --> H["Power-cycle the host"] + F --> H + H --> I["Read BMC, ERoT/CEC, and NIC inventory"] + I --> J{"All versions match the DPU model baseline?"} + J -- No --> K["Wait and retry verification"] + J -- Yes --> G + G --> L["Wait for network configuration and health"] + L --> M["Clear reprovisioning request and return host to service"] +``` + +`dpu_config.dpu_enable_secure_boot` controls the classic installation branch +and defaults to `false`: + +- When it is `false`, NICo uses the UEFI HTTP network-install path. +- When it is `true`, NICo uses Redfish BFB installation only when every attached + DPU reports BMC firmware `24.10` or newer. If one DPU does not meet that + requirement or has no reported BMC version, NICo uses network installation. +- DPF-managed provisioning takes precedence and does not use this classic + secure-boot decision. + +This setting chooses how NICo applies the BFB. It does not enable automatic DPU +selection and does not choose the BFB or component target versions. + +For the classic path, version verification uses exact string equality. NICo +checks BMC, ERoT/CEC, and NIC in that order. It waits when an inventory entry +is missing, has no version, or reports a different version. DPU UEFI is not in +this verification loop. + +For DPF-managed DPUs, Machine Update Manager asks DPF for devices whose +installed BFB or DPU flavor differs from the active `DPUDeployment`. DPF drift +can therefore trigger the same host-level scheduling flow without consulting +the classic NIC accepted-version list. DPF owns reprovisioning, and NICo skips +the classic Redfish component comparison afterward. + +## Assigned hosts and operator requests + +An assigned host is not eligible for automatic DPU NIC selection. Its drift +contributes to the unavailable count until the host returns to top-level +`Ready`. + +An operator can [request DPU reprovisioning +explicitly](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/dpu-reprovision/reprovision-machine-dpu). +The API requires a `HostUpdateInProgress` health alert with +`PreventAllocations`; the CLI's `--update-message` option creates that alert +before sending the request: + +```bash +nico-admin-cli -a dpu reprovision set \ + --id \ + --update-message "" +``` + +Passing a host ID requests reprovisioning for all attached DPUs. Passing a DPU +ID requests it only for that DPU. On an unassigned `Ready` host, the controller +can start the request directly. On an assigned host, the request waits for +tenant reboot approval or the configured `instance_autoreboot_period`, as +described in [Assigned hosts need reboot +approval](host-firmware.md#assigned-hosts-need-reboot-approval). + +The CLI still accepts `--update-firmware`, but the current DPU state machine +does not branch on that value. Every classic DPU reprovision follows the same +BFB installation or network-install path and firmware verification. + +## Monitor an update + +Show the versions reported for all DPUs: + +```bash +nico-admin-cli -a dpu versions +``` + +The output includes DPU NIC, BMC, BIOS/UEFI, HBN, and DPU agent versions. Do +not use `dpu versions --updates-only` as the authoritative drift list: the +current implementation reads an empty legacy expected-version map and can +return all DPUs instead of only drifted ones. + +List pending and active DPU reprovisioning requests, then inspect the +managed-host substate for a particular host: + +```bash +nico-admin-cli -a dpu reprovision list +nico-admin-cli -a managed-host show +``` + +The following metrics provide the site-wide view: + +| Metric | Meaning | +|---|---| +| `carbide_pending_dpu_nic_firmware_update_count` | Drifted DPUs attached to hosts currently eligible for automatic selection. | +| `carbide_unavailable_dpu_nic_firmware_update_count` | Drifted classic DPUs whose host is not currently eligible. | +| `carbide_running_dpu_updates_count` | DPU reprovisioning requests marked as firmware updates that have started. | +| `carbide_dpu_firmware_version_count{firmware_version="..."}` | DPUs grouped by the firmware version reported in their DPU hardware information. | +| `carbide_firmware_updates_total{target="dpu_nic",phase="started"}` | Automatic DPU NIC update starts. This target does not emit a `completed` phase. | +| `carbide_firmware_update_failures_total{target="dpu_nic",cause="wrong_version_after_update"}` | Reprovisioning returned to `Ready`, but the reported NIC version is still outside the accepted list. | + +## Recover a stalled update + +Start with `dpu reprovision list` and `managed-host show`. The substate usually +separates an installation problem from a post-install health or configuration +problem. + +| Symptom | Check | +|---|---| +| `InstallDpuOs/WaitForInstallComplete` does not advance | Inspect the Redfish task and `nico-api` logs. Verify that the DPU BMC can reach `carbide-pxe.forge` and that `forge.bfb` is available. | +| `WaitingForNetworkInstall` does not advance | Check the DPU's UEFI HTTP boot, PXE requests in `nico-pxe`, and whether the DPU agent starts from the installed image. | +| `VerifyFirmareVersions` keeps waiting | Compare BMC, ERoT/CEC, and NIC Redfish inventory with `dpu_models`. A mismatch usually means that the deployed BFB and configured baseline do not describe the same release. | +| `WaitingForNetworkConfig` does not advance | Check DPU agent health and whether the managed-host network configuration versions are synchronized. | +| The host returned to `Ready`, but the update marker remains | Check the NIC version against `dpu_nic_firmware_update_versions`. NICo deliberately retains the update health marker when the resulting NIC version is not accepted. | + +Restart an active or failed reprovisioning flow for all requested DPUs on a +host: + +```bash +nico-admin-cli -a dpu reprovision restart \ + --id +``` + +Clear a request only when it has not started: + +```bash +nico-admin-cli -a dpu reprovision clear \ + --id +``` + +For DPU OS installation, networking, and health details beyond firmware, see +[DPU lifecycle management](../../dpu-management/dpu-lifecycle-management.md). + +## Current limitations + +- `dpu_nic_firmware_initial_update_enabled` does not currently suppress BFB + installation or firmware verification during initial DPU setup; the machine + controller only logs the value. +- The machine controller stores + `dpu_nic_firmware_reprovision_update_enabled`, but does not consult that field + while executing a reprovisioning request. The setting controls automatic + selection through Machine Update Manager. +- The `update_firmware` value stored on a DPU reprovisioning request is used by + status and metrics, but does not select a different execution path. +- Automatic completion emits no `carbide_firmware_updates_total` event with + `phase="completed"` for the `dpu_nic` target. Use reprovisioning state, the + running gauge, and resulting inventory for completion monitoring. +- Classic post-reprovision verification does not compare `DPU_UEFI`, even + though the DPU model catalog contains a UEFI entry. diff --git a/docs/operations/firmware-updates/host-firmware.md b/docs/operations/firmware-updates/host-firmware.md new file mode 100644 index 0000000000..608e1d1555 --- /dev/null +++ b/docs/operations/firmware-updates/host-firmware.md @@ -0,0 +1,382 @@ +# Managed host firmware upgrades + +Managed host firmware upgrades keep an ingested host aligned with the default +versions in the [effective host firmware +catalog](configuration.md#host-firmware-catalog). They are normally triggered +by drift, scheduled by Machine Update Manager, and executed by the managed-host +state machine. + +This is different from [pre-ingestion](pre-ingestion.md). Pre-ingestion uses a +minimum-version threshold before the host is managed. The managed-host path +compares a managed host with its steady-state defaults and performs the work as +host reprovisioning. + +Three parts of NICo cooperate in this workflow: + +| Responsibility | NICo component | +|---|---| +| Determine whether reported firmware differs from the defaults. | Site Explorer and the host firmware catalog. | +| Decide whether another host may start an update. | Machine Update Manager. | +| Install, activate, and verify each component. | Managed-host state machine, with Redfish, Scout, or a local script. | + +## How an automatic update starts + +```mermaid +flowchart TD + A["Configured desired versions"] --> C["Compare versions"] + B["Latest reported versions"] --> C + C --> D{"Exact match?"} + D -- Yes --> Z["No update request"] + D -- No --> E{"Automatic update enabled for this host?"} + E -- No --> Z + E -- Yes --> F{"Explicit start required?"} + F -- No --> G{"Site concurrency budget available?"} + F -- Yes --> V{"Per-host update window active?"} + V -- No --> W["Keep pending"] + V -- Yes --> G + G -- No --> W + G -- Yes --> H["Create host reprovisioning request"] + H --> I{"Host has an assigned instance?"} + I -- No --> K["Begin managed-host firmware workflow"] + I -- Yes --> J{"Tenant approval or autoreboot period active?"} + J -- No --> W + J -- Yes --> L["Boot discovery environment"] + L --> K +``` + +### Drift detection + +Machine Update Manager snapshots the default version of each configured +component by vendor and model. Site Explorer independently stores the versions +reported by the host's matching Redfish inventory entries. A host is drifted +when these two normalized `Versions` maps are not exactly equal. + +This comparison is equality-based, not an ordering comparison. A version is +drifted whether it is older, newer, or simply formatted differently from the +configured default. `preingest_upgrade_when_below` is not used after ingestion. + +The catalog can change through the [Host Firmware Config +API](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/host-firmware-config/create-or-update-host-firmware-config) +or legacy/static configuration. Machine Update Manager refreshes its +desired-version snapshot when the runtime configuration changes or the legacy +firmware directory is modified. + +### Scheduling gates + +A drifted host becomes eligible only after the relevant gates pass: + +| Gate | Behavior | +|---|---| +| Automatic update policy | `firmware_global.autoupdate` supplies the site default. A per-machine override can enable or disable one host independently. | +| Host state | With `firmware_global.instance_updates_manual_tagging = true`, automatic selection is limited to unassigned hosts in `Ready`. When false, assigned hosts can also be selected. Unassigned hosts are considered first. | +| Existing work | A host with an existing reprovisioning request is not selected again. Machines already in maintenance or another update workflow count against the shared budget. | +| Explicit start | When the model has `explicit_start_needed = true`, the current time must be inside that machine's firmware update window. | +| Site capacity | `machine_updater.max_concurrent_machine_updates_absolute` defines the usable site-wide update ceiling. Machines already in maintenance consume this budget. | + +Automatic machine updates do not start when both concurrency limits are +unset, because the resulting update ceiling is zero. Machine Update Manager +runs every `machine_update_run_interval` seconds; the default is 300 seconds. + +> **Current limitation:** +> `machine_updater.max_concurrent_machine_updates_percent` is intended to set a +> percentage-based ceiling and account for unhealthy hosts. The current +> implementation passes the total and unhealthy host counts to that calculation +> in reverse, so setting the percentage option reduces the calculated ceiling +> to zero. Until that is corrected, configure +> `machine_updater.max_concurrent_machine_updates_absolute` and leave the +> percentage option unset. + +When a host is selected, NICo creates a host reprovisioning request. The +managed-host state machine then adds the `host-fw-update` health report. This +report prevents a currently unassigned host from being allocated and suppresses +external health alerting caused by expected update operations. + +### Enable or disable one host + +The per-machine policy overrides the global setting: + +```bash +nico-admin-cli -a machine auto-update \ + --machine --enable + +nico-admin-cli -a machine auto-update \ + --machine --disable + +nico-admin-cli -a machine auto-update \ + --machine --clear +``` + +`--clear` returns the host to the `firmware_global.autoupdate` policy. + +The static `firmware_global.host_enable_autoupdate` and +`firmware_global.host_disable_autoupdate` fields are legacy controls. The +current scheduler does not read `host_enable_autoupdate`, and it compares +`host_disable_autoupdate` entries only with machine IDs. A matching disabled +entry also ends host selection for that scheduler pass. Use the per-machine +command for new operational workflows. + +### Open an explicit-start window + +For a model with `explicit_start_needed = true`, set a window on the affected +machines: + +```bash +nico-admin-cli -a managed-host start-updates \ + --machines \ + --start \ + --end +``` + +Use `YYYY-MM-DDTHH:MM:SS+0000` for an explicit UTC offset, or +`YYYY-MM-DDTHH:MM:SS` for local time. +Omitting `--start` starts the window now. Omitting `--end` makes it last 24 +hours. Cancel a pending window with: + +```bash +nico-admin-cli -a managed-host start-updates \ + --machines --cancel +``` + +This command opens a scheduler gate; it does not create an update request by +itself. Models without `explicit_start_needed` do not require this window. + +### Assigned hosts need reboot approval + +An assigned host can have a pending reprovisioning request without immediately +interrupting its instance. The request starts when either: + +- the tenant approves updates while rebooting the instance; or +- the current time is inside the site-level + `machine_updater.instance_autoreboot_period`. + +Tenant approval is expressed as: + +```bash +nico-admin-cli -a instance reboot \ + --instance --apply-updates-on-reboot +``` + +The autoreboot period is separate from the per-machine firmware window. The +firmware window lets Machine Update Manager create the request for a model that +requires explicit start. The autoreboot period lets an assigned host consume an +existing request without waiting for tenant approval. + +After approval, NICo moves the assigned host through its platform configuration +and discovery-boot states before entering the same component-update loop used +for an unassigned host. `HostReprovision` is the internal state name; this is +not a return to pre-ingestion. + +### Request host reprovisioning directly + +An operator can bypass automatic drift selection and scheduling by creating a +host reprovisioning request: + +```bash +nico-admin-cli -a host reprovision set \ + --id \ + --update-message "" +``` + +The state machine still compares inventory with the catalog and returns the +host to its prior steady state if nothing needs an update. A direct request is +not constrained by Machine Update Manager's concurrency calculation, although +an assigned host still needs reboot approval. Use it only as part of an +operator-controlled maintenance procedure. + +The host reprovisioning command always enters firmware checking; its legacy +`--update-firmware` flag is not needed. A request that has not started can be +removed with `host reprovision clear --id `. + +## Upgrade sequence + +After the request can proceed, the managed-host state machine performs the +following sequence: + +1. **Protect the host.** NICo records the update health report and, where + needed, moves an assigned host into its discovery environment and disables + platform lockdown. +2. **Refresh inventory.** Site Explorer revisits the BMC. The state machine + selects the effective catalog definition by BMC vendor and host model. +3. **Select a component.** NICo walks `ordering` and finds the first component + for which any matching inventory entry differs from the default version. + Pre-ingestion-only firmware entries are excluded. +4. **Install it.** NICo chooses the installation route described below. + Multi-artifact entries are applied in sequence. +5. **Activate and verify.** For Scout and Redfish updates, NICo performs + component-specific resets, reboots, or configured power drains. A legacy + local script owns its own activation procedure. NICo then requests fresh + inventory and checks the reported versions. +6. **Repeat.** NICo returns to the beginning of `ordering` and selects the next + mismatched component. +7. **Restore service.** When no mismatch remains, NICo restores lockdown when + required, returns the host to `Ready` or its assigned-instance state, clears + the reprovisioning request, and removes the update health report. + +### Installation route + +The installation mechanism is selected per vendor, model, and component. It is +not a separate top-level workflow. + +```mermaid +flowchart TD + A["Ordered component differs from default"] --> B{"Packaged Scout script exists?"} + B -- Yes --> C["Send verified task and artifacts to Scout"] + B -- No --> D{"Firmware entry has legacy script?"} + D -- Yes --> E["Run local script from NICo Core"] + D -- No --> F["Download artifact and upload through Redfish"] + C --> G["Collect Scout result"] + E --> J["Refresh inventory and recheck component"] + F --> H["Poll Redfish task"] + H --> G + G --> I["Run NICo activation and verify through fresh inventory"] +``` + +The route precedence is: + +1. **Scout script.** NICo first looks for a packaged + `/opt/carbide/scout-firmware-scripts////upgrade.sh` + and its `metadata.toml`. If present, NICo sends Scout the verified script, + artifact URLs and digests, and execution timeouts. Scout downloads and + verifies the files, runs the script on the host, and reports the result. +2. **Core-local script.** If no Scout script exists and the firmware entry has + `script`, NICo runs that legacy script on the Core service with the BMC + address and credentials in its environment. The script is responsible for + installation and activation; after it succeeds, NICo returns to inventory + checking. +3. **Redfish.** Otherwise NICo resolves the configured artifact, waits for a + shared `firmware_global.max_uploads` slot, uploads asynchronously, and polls + the returned Redfish task. + +All successful routes eventually return to component selection. Scout and +Redfish updates first pass through NICo's activation and inventory-verification +states; the legacy script path returns directly to inventory checking. + +### Platform and OEM procedures + +NICo selects the configured component, starts the integrated installation +route, and verifies the resulting inventory. The approved platform or OEM +procedure remains authoritative for package selection, prerequisites, +preservation settings, activation requirements, and recovery. + +Before adding an artifact or script to the catalog, verify that it implements +the procedure for the exact platform and release. A successful Redfish task or +script exit only shows that the installation mechanism completed; NICo still +needs fresh inventory to confirm the configured version. See the [Redfish +workflow](../../architecture/redfish_workflow.md) for the request and task +ownership boundaries. + +### Manual platform gate + +When `firmware_global.requires_manual_upgrade` is enabled for an MNNVL-capable +host, NICo pauses in `WaitingForManualUpgrade` before selecting components. +GB200 is the common example. Complete the approved platform procedure, then +confirm it with: + +```bash +nico-admin-cli -a host reprovision \ + mark-manual-upgrade-complete --id +``` + +NICo then refreshes inventory and runs the normal component loop for anything +that still differs from the catalog. Marking the manual work complete does not +assert that every component is compliant; verification still comes from +discovered inventory. + +## Monitor and verify + +List pending or active host reprovisioning requests: + +```bash +nico-admin-cli -a host reprovision list +``` + +Inspect the current state and recent state history: + +```bash +nico-admin-cli -a managed-host show +nico-admin-cli -a machine show --history-count 10 +``` + +Use the host BMC address to inspect the firmware inventory last stored by Site +Explorer: + +```bash +nico-admin-cli -a site-explorer get-report \ + managed-host +``` + +The principal host firmware substates are: + +| State | Meaning | +|---|---| +| `CheckingFirmwareV2`, `CheckingFirmwareRepeatV2` | Refresh inventory and select the next mismatched component. | +| `WaitingForManualUpgrade` | Wait for completion of the approved platform procedure. | +| `WaitingForScoutUpgrade` | Wait for Scout to execute the packaged upgrade task and report its result. | +| `WaitingForScript` | Wait for the legacy Core-local script. | +| `WaitingForUpload` | A Redfish upload is running asynchronously. | +| `WaitingForFirmwareUpgrade` | The upload completed and NICo is polling the Redfish task. | +| `ResetForNewFirmware` | NICo is performing activation resets, reboots, or power drains. | +| `NewFirmwareReportedWait` | NICo is refreshing inventory until all matching entries report the target. | +| `FailedFirmwareUpgrade` | The attempt failed; NICo is waiting to retry or has exhausted its retry budget. | + +See the [managed-host state machine](../../architecture/state_machines/managedhost.md) +for these firmware substates in the complete host lifecycle. + +Completion requires more than leaving an installation state. Verify that the +host returned to `Ready` or `Assigned/Ready`, the reprovisioning request is no +longer listed, and Site Explorer reports the configured default versions. + +Useful metrics are: + +| Metric | What it shows | +|---|---| +| `carbide_pending_host_firmware_update_count` | Hosts currently selected by the drift query as needing firmware work. | +| `carbide_active_host_firmware_update_count` | Hosts whose top-level state is `HostReprovision`. Assigned-host firmware substates are not included by this query. | +| `carbide_exhausted_reprovision_retry_count` | Unassigned hosts whose top-level firmware retry budget is exhausted. Assigned-host update substates are not included. | +| `carbide_firmware_updates_total{target="host"}` | Host updates started and completed, separated by the `phase` label. | +| `carbide_host_reprovision_retries_total` | Automatic retries of failed host firmware reprovisioning. | +| `carbide_machines_in_maintenance_count` | Machines consuming the shared maintenance budget. | +| `carbide_concurrent_machine_updates_available` | The calculated site-wide concurrency ceiling for machine updates. | + +See [Core metrics](../../observability/core_metrics.md) for the common metric +labels and collection endpoint. + +## Failures and recovery + +Transient upload failures and interrupted asynchronous uploads return to +firmware checking and are attempted again. A failed Scout result, failed local +script, terminal Redfish task, or firmware that exhausts its activation-reset +retries moves the host to `FailedFirmwareUpgrade`. + +NICo waits `firmware_global.host_firmware_upgrade_retry_interval` before +retrying that workflow. The default interval is 60 minutes. An unassigned host +in the top-level `HostReprovision` state has a budget of five retries. After it +is exhausted, the host remains failed and is counted by +`carbide_exhausted_reprovision_retry_count`. + +> **Current limitation:** Assigned-host firmware work stores its substate under +> `Assigned` without a persisted host retry counter. It therefore retries at +> the configured interval without reaching the five-retry limit or the +> exhausted-retry metric. + +Correct the artifact, script, BMC, inventory matcher, or platform condition +before forcing another attempt. Then reset host reprovisioning to firmware +checking and, for an unassigned host, clear the retry count: + +```bash +nico-admin-cli -a managed-host \ + reset-host-reprovisioning --machine +``` + +This command does not repair the platform or change the catalog. It only asks +the state machine to start firmware checking again. + +A Core-local script interrupted by a NICo restart is not resumed automatically +and can remain in `WaitingForScript`; use the same reset command after verifying +that the old script is no longer running. + +If a host repeatedly appears as pending but immediately leaves the firmware +workflow, compare the catalog's `current_version_reported_as` matcher with the +latest Redfish inventory. A missing matching inventory entry makes the desired +and reported `Versions` maps differ, but leaves the execution path without a +component it can install or verify. diff --git a/docs/operations/firmware-updates/pre-ingestion.md b/docs/operations/firmware-updates/pre-ingestion.md new file mode 100644 index 0000000000..8801ef539e --- /dev/null +++ b/docs/operations/firmware-updates/pre-ingestion.md @@ -0,0 +1,184 @@ +# Pre-ingestion firmware upgrades + +Pre-ingestion firmware upgrades run against a discovered host BMC before NICo +creates the managed host. Their purpose is to update firmware that is too old +for the normal ingestion and management workflow. + +This path is driven by the pre-ingestion manager. It does not use Machine +Update Manager, host update windows, `explicit_start_needed`, or per-host +automatic-update overrides. A machine ID might not exist yet, so the only +automatic-update switch used by this path is `firmware_global.autoupdate`. + +This page covers host firmware selected from the [host firmware +catalog](configuration.md#host-firmware-catalog). DPU BFB recovery also runs in +the pre-ingestion state machine, but installs a DPU boot image to recover the +DPU; it is not a host firmware-catalog workflow. + +## When this path applies + +A host firmware update starts during pre-ingestion when all of the following +are true: + +1. Site exploration has produced a BMC vendor, host model, and firmware + inventory for an endpoint that has not completed pre-ingestion. +2. The effective host firmware catalog contains a definition for that vendor + and model. +3. At least one reported component has + `preingest_upgrade_when_below` configured and its reported version is lower + than that threshold. +4. `firmware_global.autoupdate` is enabled. + +The reported version must come from an inventory entry matched by +`current_version_reported_as`. A missing catalog definition, missing inventory +match, or missing version cannot trigger the update. If no other component +triggers it, NICo completes the firmware portion of pre-ingestion and continues +with ingestion. + +When `firmware_global.autoupdate` is disabled, NICo still checks for versions +below the threshold, but it does not install firmware or block ingestion. It +marks pre-ingestion complete. The host model and machine ID allowlists used by +managed-host updates are not consulted here. + +## Trigger and target + +`preingest_upgrade_when_below` decides whether firmware work starts; it does not +specify the version NICo installs. NICo selects an installable entry from +`known_firmware` that is marked `preingestion_exclusive_config` or `default`. +A pre-ingestion-only entry allows the ingestion target to differ from the +steady-state default. + +Once any component is below its threshold, NICo walks the model's `ordering` +and updates every ordered component whose reported version differs from its +pre-ingestion target. A component does not need to be below its own threshold +at this point. If no ordering is configured, the legacy default is BMC followed +by UEFI. + +For example: + +| Component | Minimum | Target | Reported on host A | Reported on host B | +|---|---|---|---|---| +| BMC | `7.00` | `7.10` | `6.50` | `7.05` | +| UEFI | `1.50` | `1.80` | `1.60` | `1.60` | + +- Host A triggers pre-ingestion firmware work because its BMC is below `7.00`. + NICo updates both BMC and UEFI because both differ from their targets. +- Host B does not trigger pre-ingestion firmware work because both components + meet their minimums. After ingestion, both can still be treated as drifted + from their steady-state targets. + +## Upgrade sequence + +```mermaid +flowchart TD + A["Fresh BMC exploration report"] --> B["BMC reset and time checks"] + B --> C{"Any reported component below its minimum?"} + C -- No --> J["Complete pre-ingestion"] + C -- Yes --> D{"Global autoupdate enabled?"} + D -- No --> J + D -- Yes --> E["Select next mismatched component in ordering"] + E --> F["Run upgrade script or upload artifacts through Redfish"] + F --> G["Wait for installation and perform activation resets"] + G --> H{"Fresh inventory reports the target?"} + H -- No --> K["Wait for inventory or retry activation"] + K --> H + H -- Yes --> I{"Another ordered component needs an update?"} + I -- Yes --> E + I -- No --> J +``` + +The state machine performs the following work: + +1. **Prepare the BMC.** NICo attempts an initial BMC reset, waits for a fresh + exploration report, configures the site's NTP servers, and checks BMC time. + Firmware comparison uses the refreshed inventory. +2. **Evaluate minimums.** NICo selects the effective catalog definition by BMC + vendor and host model, then compares matched inventory versions with each + configured pre-ingestion threshold. +3. **Select one component.** If a minimum is breached, NICo follows `ordering` + and selects the first component whose reported version differs from its + pre-ingestion target. Components are handled serially for each endpoint. +4. **Install the firmware.** A firmware entry with `script` runs the configured + local upgrade script. Otherwise NICo resolves the artifact, waits for a + shared upload slot, and uploads it through Redfish. Multi-artifact entries + are installed in sequence. +5. **Activate and verify.** NICo waits for the Redfish task, performs the + component-specific reset, reboot, or configured power drains, and requests + a fresh exploration report. When the matching inventory entry is present, + NICo waits until its reported version matches the target. A missing entry + cannot be verified or selected for another update. +6. **Continue ingestion.** NICo repeats target selection for the remaining + ordered components. When none need an update, it marks pre-ingestion + complete and the normal ingestion process can continue. + +`firmware_global.max_uploads` limits concurrent uploads across firmware +workflows. `firmware_global.concurrency_limit` limits how many endpoints the +pre-ingestion manager processes concurrently; it does not allow parallel +component updates on one endpoint. + +## Monitor progress + +Use the BMC address to inspect the latest exploration report and pre-ingestion +state: + +```bash +nico-admin-cli -a site-explorer get-report endpoint +``` + +The states most relevant to host firmware are: + +| State | Meaning | +|---|---| +| `InitialBMCReset`, `SetNtpServers`, `TimeSyncReset` | NICo is preparing the BMC and obtaining reliable inventory before comparing versions. | +| `UpgradeFirmwareWait` | The firmware was uploaded and NICo is polling the Redfish task. | +| `ScriptRunning` | The configured firmware upgrade script is running. | +| `ResetForNewFirmware` | Installation completed and NICo is performing activation resets or power drains. | +| `NewFirmwareReportedWait` | NICo is waiting for refreshed inventory to report the target version. | +| `RecheckVersions`, `RecheckVersionsAfterFailure` | NICo is checking the current component again or selecting the next one. | +| `Complete` | No applicable pre-ingestion update remains, or firmware autoupdate was disabled. Ingestion can continue. | +| `Failed` | Pre-ingestion stopped and requires operator action. The state includes the reason. | + +`Complete` does not mean every component matches its steady-state default. It +means no pre-ingestion update remains after applying the rules above. + +The following metrics provide site-wide visibility: + +| Metric | What it shows | +|---|---| +| `carbide_preingestion_total` | Endpoints currently being evaluated by the full pre-ingestion state machine. | +| `carbide_preingestion_waiting_download` | Endpoints whose firmware work was deferred because no upload slot was available. | +| `carbide_preingestion_waiting_installation` | Endpoints waiting for a Redfish firmware task to complete. | +| `carbide_preingestion_firmware_upload_total` | Upload attempts by method and outcome. | +| `carbide_preingestion_firmware_upgrade_tasks_total` | Completed and failed Redfish tasks by component and final task state. | +| `carbide_preingestion_power_control_total` | Host, BMC, and chassis power operations by operation and outcome. | + +## Failures and retries + +NICo retries most transient conditions on later pre-ingestion passes. These +include an unavailable BMC, a busy upload limit, artifact download still in +progress, Redfish upload errors, and inventory that has not yet refreshed. A +failed Redfish task causes NICo to refresh inventory and evaluate the component +again. If an installed version is still not reported after 30 minutes, NICo can +repeat the activation reset unless `firmware_global.no_reset_retries` is set. + +Some failures, including an unsuccessful upgrade script or exhausted BMC time +synchronization attempts, move the endpoint to `Failed`. Correct the underlying +problem, then clear the error to restart pre-ingestion from `Initial`: + +```bash +nico-admin-cli -a site-explorer clear-error +``` + +The command resets only a terminal `Failed` pre-ingestion state; it does not +interrupt an update that is still progressing. + +An upgrade script interrupted by a NICo restart is not resumed automatically +and can remain in `ScriptRunning`. After confirming that the script and firmware +operation are no longer running, an operator can delete the unpaired explored +endpoint as a last resort and allow it to be discovered again: + +```bash +nico-admin-cli -a site-explorer delete --address +``` + +NICo rejects explored-endpoint deletion after the endpoint belongs to a managed +host. diff --git a/docs/operations/firmware-updates/rack-component-firmware.md b/docs/operations/firmware-updates/rack-component-firmware.md new file mode 100644 index 0000000000..c5f2b5ef91 --- /dev/null +++ b/docs/operations/firmware-updates/rack-component-firmware.md @@ -0,0 +1,332 @@ +# Rack and tray firmware upgrades + +Firmware updates for racks, compute trays, NVSwitches, and power shelves are +explicit operations. NICo does not start them from the host or DPU drift +detectors described elsewhere in this guide. A Provider Admin submits a REST +request, and NICo Flow creates one or more asynchronous tasks to sequence the +work. + +Use this path when an operator needs Flow to update a rack or selected tray. +For desired-state updates on a conventional managed host, use [Host firmware +upgrades](host-firmware.md) instead. + +## How the request runs + +The REST response confirms that the task was created; it does not mean that +the firmware update has completed. + +```mermaid +flowchart LR + A["Provider Admin submits rack or tray request"] --> B["Flow resolves targets and an operation rule"] + B --> C["Wait for affected hosts to be ready"] + C --> D["Start update through NICo Core"] + D --> E["Component backend applies firmware"] + E --> F["Flow polls component status"] + F --> G{"Terminal result?"} + G -- No --> F + G -- Success --> H["Task succeeds"] + G -- Failure or timeout --> I["Task fails with a report"] +``` + +Before dispatching a disruptive operation, Flow checks persisted component +readiness. A compute-tray update checks the targeted hosts. An NVSwitch or +power-shelf update checks the hosts in the owning rack, because those devices +can affect every tenant in that rack. Flow waits for up to 30 minutes and +checks every 5 seconds by default. + +If readiness information is missing, the current implementation logs a warning +and allows the operation to proceed. Do not treat the readiness check as a +substitute for confirming the maintenance scope and tenant impact. + +## Choose the endpoint + +All endpoints require the Provider Admin role. `siteId` is required in every +request body and must identify a site owned by the provider organization. + +| Scope | Endpoint | Selection | +|---|---|---| +| One rack | [`PATCH /v2/org/{org}/nico/rack/{rack-id}/firmware`](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/rack/firmware-update-rack) | The rack in the URL. | +| Several or all racks | [`PATCH /v2/org/{org}/nico/rack/firmware`](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/rack/firmware-update-racks) | Optional `filter.names`. No filter means every rack in the site. | +| One tray | [`PATCH /v2/org/{org}/nico/tray/{tray-id}/firmware`](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/tray/firmware-update-tray) | The tray in the URL. | +| Several or all trays | [`PATCH /v2/org/{org}/nico/tray/firmware`](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/tray/firmware-update-trays) | Optional `filter`. No filter means all compute, NVSwitch, and power-shelf trays in every site rack. | + +The batch tray filter supports: + +| Filter | Meaning | +|---|---| +| `rackId` or `rackName` | Select trays in one rack. These fields are mutually exclusive. | +| `type` | Select exactly `Compute`, `NVSwitch`, or `PowerShelf`. | +| `componentIds` | Select component IDs of the specified `type`; `type` is required. | +| `ids` | Select tray UUIDs. | +| `slotId` | Select a rack slot; requires `rackId` or `rackName`. | + +Rack selection cannot be combined with `ids` or `componentIds`. Prefer a +single-rack or single-tray request for the first update of a new firmware +bundle. An unfiltered batch request can affect the entire site. + +## Describe the update + +The request has four controls in addition to `siteId`: + +| Field | Purpose | +|---|---| +| `version` | Target passed to the component backend. For current rack-scale RMS paths, this is a complete SOT firmware-object JSON document serialized as a string. Legacy backends can accept a plain version string. | +| `targets` | Optional component subset for tray requests. When present, `version` must also be present. Rack handlers do not forward this field, so do not send it with a rack request. | +| `ruleId` | Pins the task to a custom Flow operation rule. When omitted, Flow resolves a rule and falls back to its built-in firmware rule. | +| `overrideReadinessCheck` | Bypasses Flow's readiness gate and tells Core to bypass its state controller where supported. Use only during supervised maintenance after tenant impact has been accepted. | + +Although the API permits `version` to be omitted when `targets` is empty, that +is not portable across component backends. Rack-scale RMS updates require SOT +JSON. Supply an explicit version unless the selected backend and operation rule +are known to resolve one. + +### SOT firmware-object JSON + +RMS firmware is described by a source-of-truth (SOT) firmware object: a JSON +document containing the bundle identity and the artifacts RMS must apply. The +document comes from the platform's firmware release process; it is not the +same as the host firmware catalog described in [Configure firmware +versions](configuration.md). + +The REST `version` field is a string, so the complete JSON document must be +serialized into that string. Build the request body with a JSON tool instead +of escaping the document by hand: + +```sh +SOT_JSON=$(jq -c . compute-firmware-object.json) + +jq -n \ + --arg siteId "$SITE_ID" \ + --arg version "$SOT_JSON" \ + '{siteId: $siteId, version: $version, targets: ["bmc", "bios"]}' +``` + +For a rack request that needs a different value for each component type, +`version` can contain a layered JSON document with `compute`, `nvswitch`, and +`powershelf` keys. Flow extracts the relevant value before calling each +component manager. For example, this builds a layered value from two complete +SOT documents: + +```sh +COMPUTE_SOT=$(jq -c . compute-firmware-object.json) +SWITCH_SOT=$(jq -c . switch-firmware-object.json) + +LAYERED_VERSION=$(jq -cn \ + --argjson compute "$COMPUTE_SOT" \ + --argjson nvswitch "$SWITCH_SOT" \ + '{compute: $compute, nvswitch: $nvswitch}') +``` + +If a layered document omits a component-type key, Flow passes an empty target +to that component manager. Use an operation rule that excludes the component +instead of relying on an empty value to skip it. + +The REST surface does not accept an RMS artifact access token. Core sends +`NOAUTH` when no token is available, so artifacts referenced by these requests +must be reachable without a separate token. + +## Submit an update + +This request updates only the BMC and BIOS targets on one compute tray: + +```json +{ + "siteId": "2b88bb63-9a21-4bad-b113-68a54aa6e3dd", + "version": "", + "targets": ["bmc", "bios"] +} +``` + +The response contains the task IDs to monitor: + +```json +{ + "taskIds": ["c5e00a88-9b42-4e2b-a237-63e787f698ef"] +} +``` + +To update all racks named `A01` and `A02`, submit a batch rack request: + +```json +{ + "siteId": "2b88bb63-9a21-4bad-b113-68a54aa6e3dd", + "filter": { + "names": ["A01", "A02"] + }, + "version": "" +} +``` + +## Default sequencing + +When no site-specific rule or `ruleId` applies, Flow uses this built-in rule: + +| Stage | Component type | Status polling | Attempts | +|---|---|---|---| +| 1 | Compute | Every 2 minutes, for up to 45 minutes | 2 | +| 2 | NVSwitch | Every 2 minutes, for up to 45 minutes | 2 | + +The stages run in order. A failed or timed-out stage fails the task. A stage +that has no matching components is reported as skipped. + +The built-in rule deliberately excludes power shelves and does not perform an +AC power cycle after flashing. Use an approved custom operation rule for power +shelves. If firmware activation requires a power cycle, submit the appropriate +power-recycle task separately or include it in a custom rule. See the [Flow +operation rules guide](../../../rest-api/flow/docs/operation-rules-guide.md). + +## Component behavior + +### Compute trays + +Supported `targets` are: + +`bmc`, `bios`, `cec`, `nic`, `cpld_mb`, `cpld_pdb`, `hgx_bmc`, +`combined_bmc_uefi`, `gpu`, and `cx7`. + +Omitting `targets` asks Core to update all components represented by the +selected firmware bundle. It does not include DPU reprovisioning. + +`dpu` is a special, explicit-only compute target. NICo first submits any +compute-tray update, then reprovisions the DPU on each selected host serially. +The request's `version` is ignored for the DPU branch; its target comes from +site configuration. Use `targets: ["dpu"]` for a DPU-only request. See [DPU +firmware upgrades](dpu-firmware.md#assigned-hosts-and-operator-requests) before +using this path. + +Core routes conventional standalone machines through the managed-host firmware +workflow and rack-scale compute trays through the configured rack state +controller or component backend. Consequently, a single REST shape can start +different internal workflows depending on the hardware model. + +### NVSwitches + +Supported `targets` are `bmc`, `cpld`, `bios`, and `nvos`. Omitting `targets` +passes an empty component list to Core, which means all supported switch +components for the selected backend. + +When `version` is omitted, Flow first compares the switches' inventory with +Core's desired switch versions and skips the call if every switch is already +current. If an update is needed, the backend must still be able to resolve an +empty target. Supply a target explicitly for predictable behavior. + +### Power shelves + +Supported `targets` are `pmc` and `psu`. If `targets` is omitted, the current +Flow component manager updates only `pmc`. + +Power shelves are not present in the built-in firmware rule, so a power-shelf +tray request needs an operation rule that contains a `PowerShelf` firmware +step. Core's power-shelf state-controller path is not implemented; updates are +currently dispatched through the configured direct backend. + +## Monitor and cancel tasks + +Use [Retrieve a +Task](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/task/get-task) +to read each returned task ID until it reaches `Succeeded`, `Failed`, or +`Terminated`: + +```text +GET /v2/org/{org}/nico/task/{task-id}?siteId={site-id} +``` + +The task report records each rule stage and component step as `pending`, +`running`, `completed`, `failed`, or `skipped`. On failure, inspect both the +top-level `message` and the report's stage or step `error`. + +You can also [list tasks for a +rack](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/rack/get-rack-tasks) +or [list tasks for a +tray](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/tray/get-tray-tasks). +`activeOnly=true` restricts the result to non-terminal tasks; +`includeReport=true` includes the stage and step report. + +```text +GET /v2/org/{org}/nico/rack/{rack-id}/task?siteId={site-id}&activeOnly=true&includeReport=true +GET /v2/org/{org}/nico/tray/{tray-id}/task?siteId={site-id}&activeOnly=true&includeReport=true +``` + +[Cancel a +Task](https://docs.nvidia.com/infra-controller/rest-api-reference/api-reference/task/cancel-task) +is best effort. It terminates a pending, running, or waiting task, but it cannot +undo firmware work already accepted by a hardware backend. Completed and +failed tasks cannot be cancelled. + +```text +POST /v2/org/{org}/nico/task/{task-id}/cancel + +{"siteId":""} +``` + +After cancellation or failure, inspect component status before retrying. Do not +assume that every component remained on its previous version. + +## Inspect Core component status + +The `component-manager` commands call NICo Core directly. Use them to inspect +the backend's view of a running REST task or for platform-specific recovery. +They do not create a Flow task or apply its operation rule. + +```sh +nico-admin-cli component-manager get-firmware-update-status rack \ + --rack-id + +nico-admin-cli component-manager get-firmware-update-status compute-tray \ + --machine-id + +nico-admin-cli component-manager get-firmware-update-status switch \ + --switch-id + +nico-admin-cli component-manager get-firmware-update-status power-shelf \ + --power-shelf-id + +nico-admin-cli component-manager get-firmware-versions switch \ + --switch-id +``` + +Core also exposes direct update commands. Prefer the REST workflow for normal +rack operations because it provides readiness checks, ordering, task reports, +and cancellation. Direct commands require the operator to provide those +safeguards. + +For RMS-backed compute trays or switches, pass a SOT file rather than embedding +it on the command line: + +```sh +nico-admin-cli component-manager update-firmware compute-tray \ + --machine-id \ + --sot-json-file ./compute-firmware-object.json + +nico-admin-cli component-manager update-firmware switch \ + --switch-id \ + --sot-json-file ./switch-firmware-object.json \ + --component bmc,cpld,bios,nvos + +nico-admin-cli component-manager update-firmware power-shelf \ + --power-shelf-id \ + --target-version \ + --component pmc,psu +``` + +Run `nico-admin-cli component-manager update-firmware --help` for the complete +command options. See the [rack state +machine](../../architecture/state_machines/rackstatemachine.md) for lower-level +execution details. + +## Troubleshooting + +| Symptom | Check | +|---|---| +| No work starts after the REST response | Read the returned task. It may be waiting at the readiness gate or for an earlier rule stage. | +| Task fails after about 30 minutes | Inspect the error for component IDs blocked by the readiness gate. Confirm tenant state and the persisted component operation status. | +| Stage times out | Check Core and backend status. The built-in firmware rule polls for 45 minutes per attempt; a backend job can still be running when Flow times out. | +| Rack-scale update rejects `version` | Confirm that `version` contains a valid SOT JSON object, serialized as a string, and that referenced artifacts are reachable without a REST-supplied access token. | +| Power-shelf request succeeds without updating a shelf | Confirm that the resolved operation rule contains a `PowerShelf` step. The built-in rule excludes power shelves. | +| Firmware was flashed but is not active | Determine whether the platform requires an AC cycle. The built-in firmware rule does not include one. | +| Retry begins from an uncertain state | Inspect per-component status and inventory first. A Flow task failure or cancellation does not roll hardware back. | + +Set `overrideReadinessCheck: true` only after diagnosing a readiness block and +confirming that the affected hardware is in a supervised maintenance window. +The override bypasses a tenant-safety guard and may also send the update +directly to the component backend instead of through Core's state controller. diff --git a/docs/operations/scout-firmware-assigned-host-gap.md b/docs/operations/scout-firmware-assigned-host-gap.md new file mode 100644 index 0000000000..ba233cf4db --- /dev/null +++ b/docs/operations/scout-firmware-assigned-host-gap.md @@ -0,0 +1,142 @@ +# Scout Firmware Assigned-Host State Gap + +## Summary + +Scout-based host firmware upgrades can enter a waiting state that the Scout API +does not currently appear to serve when the host is assigned to an instance. + +This is a suspected state-shape coverage gap. The managed-host controller can +create a `WaitingForScoutUpgrade` host firmware state in two places: + +- top-level host reprovisioning for unassigned or ready hosts +- nested instance host reprovisioning for assigned hosts + +The Scout action and status-report handlers currently match only the top-level +form. If an assigned host enters the nested form, Scout may receive no firmware +upgrade action and the controller may wait until the Scout firmware deadline +expires. + +## Relevant State Shapes + +Top-level host reprovisioning: + +```rust +ManagedHostState::HostReprovision { + reprovision_state: HostReprovisionState::WaitingForScoutUpgrade { .. }, + .. +} +``` + +Assigned-host reprovisioning: + +```rust +ManagedHostState::Assigned { + instance_state: InstanceState::HostReprovision { + reprovision_state: HostReprovisionState::WaitingForScoutUpgrade { .. }, + }, +} +``` + +The state machine intentionally supports both shapes through +`HostFirmwareScenario::actual_new_state`: + +- [`crates/machine-controller/src/handler.rs`](../../crates/machine-controller/src/handler.rs) + +`HostFirmwareScenario::Ready` stores host firmware progress as top-level +`ManagedHostState::HostReprovision`. + +`HostFirmwareScenario::Instance` stores host firmware progress under +`ManagedHostState::Assigned { instance_state: InstanceState::HostReprovision }`. + +## Current Scout API Behavior + +Scout receives firmware work through `forge_agent_control`: + +- [`crates/api-core/src/handlers/machine_scout.rs`](../../crates/api-core/src/handlers/machine_scout.rs) + +That handler currently emits `Action::FirmwareUpgrade` only when the machine +state is top-level `ManagedHostState::HostReprovision` with +`HostReprovisionState::WaitingForScoutUpgrade`. + +Scout reports completion through `report_scout_firmware_upgrade_status`: + +- [`crates/api-core/src/handlers/host_reprovisioning.rs`](../../crates/api-core/src/handlers/host_reprovisioning.rs) + +That handler also destructures only the top-level host reprovisioning form and +rewrites the result back into the top-level form. + +Neither path appears to match or preserve the nested assigned-host form. + +## Expected Failure Mode + +The likely failure mode is: + +1. An assigned host is approved for host firmware upgrade. +2. The controller enters: + + ```rust + ManagedHostState::Assigned { + instance_state: InstanceState::HostReprovision { + reprovision_state: HostReprovisionState::WaitingForScoutUpgrade { .. }, + }, + } + ``` + +3. Scout polls `forge_agent_control`. +4. The API does not match the nested state and returns no firmware upgrade + action. +5. The controller continues waiting for a Scout result. +6. The `WaitingForScoutUpgrade` deadline eventually expires and the firmware + upgrade fails or retries. + +## Why This Matters + +This can affect Scout-script firmware upgrades for machines that are still +assigned to an instance and enter the assigned-host firmware reprovisioning +path. The top-level unassigned-host path is not affected by this specific state +shape mismatch. + +PXE handling already recognizes assigned host reprovisioning as a discovery +image boot case, which is another signal that assigned-host host reprovisioning +is expected to be valid: + +- [`crates/api-core/src/ipxe.rs`](../../crates/api-core/src/ipxe.rs) + +## Suggested Fix + +Teach both Scout handlers to handle both state shapes: + +- `ManagedHostState::HostReprovision { reprovision_state: WaitingForScoutUpgrade, .. }` +- `ManagedHostState::Assigned { instance_state: InstanceState::HostReprovision { reprovision_state: WaitingForScoutUpgrade } }` + +The status-report handler should preserve the original outer state shape when +writing the result: + +- top-level input should remain top-level `HostReprovision` +- assigned-host input should remain `Assigned { instance_state: + HostReprovision }` + +Avoid converting assigned-host firmware state into top-level host +reprovisioning state, because that would lose the instance lifecycle context. + +## Suggested Tests + +Add API-core tests covering: + +- `forge_agent_control` returns `Action::FirmwareUpgrade` for top-level + `WaitingForScoutUpgrade` +- `forge_agent_control` returns `Action::FirmwareUpgrade` for nested assigned + `WaitingForScoutUpgrade` +- `report_scout_firmware_upgrade_status` accepts and updates top-level + `WaitingForScoutUpgrade` +- `report_scout_firmware_upgrade_status` accepts and updates nested assigned + `WaitingForScoutUpgrade` +- stale `upgrade_task_id` is rejected in both shapes + +Existing related test targets from the Scout firmware work are useful starting +points: + +```bash +RUSTC_WRAPPER= cargo test -p carbide-api-core --test integration rejects_stale_task_id +RUSTC_WRAPPER= cargo test -p carbide-scout firmware_upgrade_status_request --bin forge-scout +```