Skip to content

[Nexthop][bugfix] Preserve accumulated port counters across admin disable/enable - #1732

Open
vvasavada-nexthop wants to merge 1 commit into
meta:mainfrom
nexthop-ai:preserve-accumulated-port-counters
Open

vvasavada-nexthop wants to merge 1 commit into
meta:mainfrom
nexthop-ai:preserve-accumulated-port-counters

Conversation

@vvasavada-nexthop

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Summary

An admin down/up of a port (fboss2 bounce interface, or fboss2 set port <p> state disable then enable) silently zeroes that port's Input Discards. Input Errors, bytes and packets keep counting through the flap.

Root cause. SaiPortManager drops a port's HwPortFb303Stats entry when the port is admin-disabled and creates a fresh one when it is re-enabled. Most port counters are read from the ASIC as cumulative values, so a fresh entry picks them up again on the first poll. A few counters are summed in software instead:

  • inDiscards_: the agent adds up changes in inDiscardsRaw_.
  • FEC correctable/uncorrectable/corrected-symbol counts, PFC duration and debounce retrigger counts: read-and-clear from hardware, then summed.

Fix.

  • On an enabled → disabled transition, save the port's last HwPortStats in a new disabledPortStats_ map. Only ports that have been polled at least once are saved.
  • On disabled → enabled, seed the new HwPortFb303Stats from the saved copy. The next poll then measures the change against the saved raw value, so discards that happened while the port was down are counted too. The summed counters stay monotonic, consistent with the hardware ones.
  • clearStats() and clearInterfacePhyCounters() also zero the saved copy. Otherwise an explicit clear of a disabled port would be undone on re-enable.
  • removePort() drops the saved copy.

Disabled ports still don't export counters, and removing and re-creating a port (lane/speed change) still starts from fresh stats.

Test Plan

New tests in PortManagerAccumulatedStatsTest:

  • preservedAcrossDisableEnable: seeds inDiscards_=1000 and fecCorrectableErrors=20, then disables
    and re-enables the port and polls. Checks that both values survive.
  • clearWhileDisabled: runs clearStats / clearInterfacePhyCounters while the port is disabled, then
    re-enables it. Checks that both counters read 0.

Run: ./switch_test --gtest_filter='PortManagerAccumulatedStatsTest.*:PortManagerTest.*'

With the fix:

  [       OK ] PortManagerAccumulatedStatsTest.preservedAcrossDisableEnable (4 ms)
  [       OK ] PortManagerAccumulatedStatsTest.clearWhileDisabled (4 ms)
  [  PASSED  ] 36 tests.

Without the fix (only the SaiPortManager changes reverted), both new tests fail. inDiscards_ reads -1
right after re-enable and 0 after the first poll, which reproduces the crow235 signature:

  [  FAILED  ] PortManagerTest.changePortNameAndCheckCounters
  [  FAILED  ] PortManagerAccumulatedStatsTest.preservedAcrossDisableEnable
  [  FAILED  ] PortManagerAccumulatedStatsTest.clearWhileDisabled
  [  PASSED  ] 34 tests.

@vvasavada-nexthop
vvasavada-nexthop requested a review from a team as a code owner October 8, 2026 17:49
@meta-cla meta-cla Bot added the CLA Signed label Oct 8, 2026
@vvasavada-nexthop vvasavada-nexthop changed the title [Nexthop] Preserve accumulated port counters across admin disable/enable [Nexthop][bugfix] Preserve accumulated port counters across admin disable/enable Oct 20, 2026
@github-actions github-actions Bot added nexthop project:bugfix FBOSS project: bugfix labels Oct 20, 2026
@vvasavada-nexthop

Copy link
Copy Markdown
Contributor Author

/autorebase

Signed-off-by: Vishrant Vasavada <vvasavada@nexthop.ai>
@rabbit-nexthop
rabbit-nexthop force-pushed the preserve-accumulated-port-counters branch from 4b710a7 to 5b53f98 Compare November 20, 2026 00:55

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant