Skip to content

[Nexthop][m4061nhp] Monitor every DCDC and NVMe channel and add warning thresholds - #2026

Open
frank-nexthop wants to merge 2 commits into
meta:mainfrom
nexthop-ai:frank.upstream.m4061nhp-sensor-coverage
Open

frank-nexthop wants to merge 2 commits into
meta:mainfrom
nexthop-ai:frank.upstream.m4061nhp-sensor-coverage

Conversation

@frank-nexthop

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Summary

Stacked on #1870, which adds the DCDCn_<RAIL>_VOUT sensors this PR adds warning bands to. The first commit is #1870's; only the second commit is new here.

On m4061nhp, platform_manager publishes many DCDC and NVMe hwmon channels that sensor_service doesn't read, and many rails have a critical band but no warning band. This PR adds 85 sensors and fills in missing limits on 35 existing ones:

  • DCDC1-10: VOUT, IOUT, POUT and power-stage temperature for each loop, plus each regulator's 12 V input, VMON pin (the 5 V power-stage bias) and controller temperature.
  • DCDC0: input voltage, output power and controller temperature. Its second loop is not populated.
  • NVMe: the two per-sensor temperatures, next to the composite.
  • Warning bands on existing rails:
    • Rails whose regulator exposes only fault limits get nominal +-5 %.
    • The AVS rails' band spans their allowed voltage range.
    • The 12 V bus uses the band programmed on the switch-card ADM1266.
  • Remaining gaps: PSU fan critical, PSU TEMP2 warning, and management temperature sensor warning.

The new IOUT warnings are each rail's maximum load current, and the POUT warnings are that current times the nominal VOUT. Criticals are the regulator's programmed fault limits. Input current and input power per loop are left out, because there is no limit to check them against.

Test Plan

On an m4061nhp, I loaded this config into sensor_service. It passed config validation, and all 217 sensors read inside their bands, with no read failures. Some sample readings (value, critical band, warning band):

DCDC0_VIN                12.25   [9.60, 13.44]   [10.80, 13.20]
DCDC0_POS0V8_VDDC_POUT  164.25   [, 1449.00]     [, 995.28]
DCDC1_POS0V9_WEST_IOUT   98.20   [, 156.00]      [, 120.00]
DCDC3_POS3V3_WEST_VOUT    3.39   [2.72, 3.90]    [2.97, 3.63]
DCDC3_POS3V3_WEST_POUT  561.00   [, 1390.96]     [, 897.60]
DCDC5_POS0V8_PT1_VOUT     0.75   [0.60, 1.03]    [0.71, 0.87]
NVME_SENSOR2_TEMP        42.85   [, 95.00]       [, 90.00]

fan_service kept all rotors healthy throughout. The JSON round-trips through json.dumps(indent=2) unchanged, with no duplicate sensor names and no existing sensor removed.

The two switch-card ADM1266 DPMs sense the ASIC rails near the source,
so under load they read 40-75 mV above the regulator output. For the 18
DPM channels whose rail is also made by a digital DCDC (DCDC0-10), drop
the DPM sensor and add a DCDCn_<RAIL>_VOUT sensor that reads the
regulator directly. That also adds the PT1/PT3/PT4/PT6 loops, which no
DPM channel watches. DCDC0 iout1 replaces the DPM VDDC IMON channel.

Thresholds mirror the limits the hwmon driver reports for each DCDC:
DCDC1-10 only expose VOUT fault limits (lcrit/crit), so those rails get
a critical band only; DCDC0 also exposes min/max, used as the alarm
band. The 3V3 limits get the same 6/5 divider scaling as the reading.

The adm1266 driver stays loaded, so the raw DPM values remain available
outside sensor_service.

Signed-off-by: frank-nexthop <frank@nexthop.ai>
platform_manager publishes many hwmon channels on m4061nhp that
sensor_service did not read, and many rails had a critical band but no
warning band. Add a sensor for every DCDC loop's VOUT, IOUT, POUT and
power-stage temperature, each DCDC's 12 V input, VMON pin and controller
temperature, DCDC0's input voltage, output power and controller
temperature, and the NVMe per-sensor temperatures.

Warning limits come from the board's rail ratings (maximum load current,
times nominal VOUT for power), the 12 V and 5 V bands programmed on the
switch-card ADM1266, and nominal +-5 % for rails whose regulator exposes
only fault limits (the AVS rails span their allowed voltage range).
Criticals stay at the regulator's programmed fault limits. Also add the
missing PSU fan critical, PSU TEMP2 warning and management temperature
sensor warning.

Signed-off-by: frank-nexthop <frank@nexthop.ai>
@frank-nexthop
frank-nexthop requested a review from a team as a code owner November 20, 2026 00:09
@github-actions github-actions Bot added nexthop project:m4061nhp FBOSS project: m4061nhp labels Nov 20, 2026
@meta-cla meta-cla Bot added the CLA Signed label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants