Skip to content

[Nexthop][fboss2-dev] Declare production features on CLI integration test fixtures - #2024

Open
hillol-nexthop wants to merge 2 commits into
meta:mainfrom
nexthop-ai:declare-production-features-on-cli
Open

hillol-nexthop wants to merge 2 commits into
meta:mainfrom
nexthop-ai:declare-production-features-on-cli

Conversation

@hillol-nexthop

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Summary

Stacked on #2023; only the last commit belongs to this PR.

Adds the first production feature declarations to the CLI integration fixtures. They are inert until run_test.py is given --enable-production-features.

Fixture split

TunnelIpInIpTest keeps the helpers and becomes the base of three fixtures:

Fixture Declares Tests
TunnelIpInIpDecapTest IP_IN_IP_DECAP the decap tests
TunnelIpInIpEncapTest nothing the encap tests and DeleteWrongDirectionRejected (it commits an encap tunnel)
TunnelIpInIpValidationTest nothing tests the CLI rejects, or treats as a no-op, before anything reaches the agent

The tunnel tests get a new suite name, so their result history restarts.

Fixture-level declarations

  • PFC: ConfigPfcTest, DeletePfcTest
  • COPP: ConfigCoppTest, ConfigCoppTrafficPolicyTest, DeleteCoppTest
  • INGRESS_MIRRORING, EGRESS_MIRRORING: ConfigInterfaceMirrorTest

Deliberately not declared

asic_production_features records what the agent tests verify per ASIC. A CLI test only needs the agent to accept and program the config, so in our hardware runs some CLI tests pass on ASICs the table does not list the matching feature for. Declaring those features would filter out passing tests, so they are left out:

Test Feature Table omits it for CLI test passes on
Encap tunnel tests IP_IN_IP_ENCAP tomahawk5, tomahawk6, g202x all three
ConfigPfcTest.RxPfcDurationEnable PFC_RX_DURATION tomahawk5, tomahawk6, g202x all three
ConfigPfcTest.TxPfcDurationEnable PFC_TX_DURATION tomahawk6, g202x both

VLAN, LAG, PTP, sFlow, MAC learning and load-balancing fixtures are also left undeclared: the table omits those features for qumran4d and we have no results to confirm the tests fail there.

Open question for reviewers

IP_IN_IP_DECAP has the same shape: the table omits it for tomahawk6, yet the decap tests pass there in our runs. With this declaration, --enable-production-features tomahawk6 filters them out. We kept the declaration to follow the table; if a CLI config test should not be gated this way, or the table should gain the entry, we can drop or adjust it.

Test plan

  • cmake build of this branch in our CI (clang): fboss2_integration_test compiles and links. The run as a whole did not finish, because fboss/agent/hw/sai/api/bcm/SwitchApi.cpp from current main does not compile against the prebuilt SAI our CI uses (SAI_SWITCH_ATTR_EXT_STATIC_FDB_REPLACE_DYNAMIC undeclared); this PR does not touch that file. The C++ unit test step did not run for the same reason.
  • In our tree, where the same change is merged, --list_production_feature on the built binary reports IP_IN_IP_DECAP for the TunnelIpInIpDecapTest tests, PFC for the PFC fixtures, COPP for the CoPP fixtures and INGRESS_MIRRORING,EGRESS_MIRRORING for ConfigInterfaceMirrorTest; every other test reports an empty list.
  • Also in our tree, Fboss2IntegrationTestRunner._filter_tests was run against the built binary and the checked-in asic_production_features JSON: tomahawk5 and g202x keep every test, tomahawk6 and qumran4d drop TunnelIpInIpDecapTest, ebro drops ConfigPfcTest.

Review Findings

Pre-publication review of the final diff found no build or correctness issues. One note: the IP_IN_IP_DECAP declaration filters tests that pass on tomahawk6, as described under "Open question for reviewers".

…n features

Let fboss2_integration_test be filtered by ASIC production features, the
way the agent, qsfp and link tests are. This change is plumbing only: no
fixture declares a feature yet, so nothing is filtered.

Test binary:

- Fboss2IntegrationTest::addProductionFeatures({...}), called from a
  fixture's constructor. A derived fixture adds to what its base
  declared.
- --list_production_feature prints "Feature List: A,B" per test, the
  same format the other test binaries use.
- In list mode the base constructor skips the test, so gtest never calls
  SetUp() or TearDown(). This is why features are declared in the
  constructor instead of a virtual getProductionFeatures(): many
  fixtures override TearDown(), several of them restore config with an
  agent restart, and a skip raised in the base SetUp() would not stop
  those from running during listing.
- ConfigVlanCreateTest::SetUpTestSuite() returns early in list mode,
  since it queries the agent.

run_test.py:

- The production feature filter (arguments, SDK gating, per-test
  listing) moves from SaiAgentTestRunner to ProductionFeatureFilterMixin
  in runners/production_features.py. sai_agent behaviour is unchanged.
- fboss2_integration uses the mixin and accepts
  --enable-production-features <asic>, --production-features <file>,
  --list-tests-for-features and --test-config-options.
…ures

Add the first production feature declarations to the CLI integration
fixtures. They are inert until run_test.py is given
--enable-production-features.

TunnelIpInIpTest keeps the helpers and becomes the base of three
fixtures:

- TunnelIpInIpDecapTest declares IP_IN_IP_DECAP and holds the decap
  tests.
- TunnelIpInIpEncapTest declares nothing and holds the encap tests,
  plus DeleteWrongDirectionRejected, which commits an encap tunnel.
- TunnelIpInIpValidationTest declares nothing and holds the tests the
  CLI rejects, or treats as a no-op, before anything reaches the agent.

The encap fixture declares nothing on purpose: asic_production_features
lists IP_IN_IP_ENCAP for one ASIC only, but the encap tests pass on
others, e.g. Tomahawk5.

Fixture-level declarations:

- PFC: ConfigPfcTest, DeletePfcTest
- COPP: ConfigCoppTest, ConfigCoppTrafficPolicyTest, DeleteCoppTest
- INGRESS_MIRRORING, EGRESS_MIRRORING: ConfigInterfaceMirrorTest

The tunnel tests get a new suite name, so their result history restarts.
@hillol-nexthop
hillol-nexthop requested review from a team as code owners November 19, 2026 09:52
@meta-cla meta-cla Bot added the CLA Signed label Nov 19, 2026
@github-actions github-actions Bot added nexthop project:fboss2-dev FBOSS project: fboss2-dev labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants