Repository navigation
[Nexthop][fboss2-dev] Declare production features on CLI integration test fixtures - #2024
Open
hillol-nexthop wants to merge 2 commits into
Open
hillol-nexthop wants to merge 2 commits into
hillol-nexthop wants to merge 2 commits into
Conversation
…n features
Let fboss2_integration_test be filtered by ASIC production features, the
way the agent, qsfp and link tests are. This change is plumbing only: no
fixture declares a feature yet, so nothing is filtered.
Test binary:
- Fboss2IntegrationTest::addProductionFeatures({...}), called from a
fixture's constructor. A derived fixture adds to what its base
declared.
- --list_production_feature prints "Feature List: A,B" per test, the
same format the other test binaries use.
- In list mode the base constructor skips the test, so gtest never calls
SetUp() or TearDown(). This is why features are declared in the
constructor instead of a virtual getProductionFeatures(): many
fixtures override TearDown(), several of them restore config with an
agent restart, and a skip raised in the base SetUp() would not stop
those from running during listing.
- ConfigVlanCreateTest::SetUpTestSuite() returns early in list mode,
since it queries the agent.
run_test.py:
- The production feature filter (arguments, SDK gating, per-test
listing) moves from SaiAgentTestRunner to ProductionFeatureFilterMixin
in runners/production_features.py. sai_agent behaviour is unchanged.
- fboss2_integration uses the mixin and accepts
--enable-production-features <asic>, --production-features <file>,
--list-tests-for-features and --test-config-options.
…ures Add the first production feature declarations to the CLI integration fixtures. They are inert until run_test.py is given --enable-production-features. TunnelIpInIpTest keeps the helpers and becomes the base of three fixtures: - TunnelIpInIpDecapTest declares IP_IN_IP_DECAP and holds the decap tests. - TunnelIpInIpEncapTest declares nothing and holds the encap tests, plus DeleteWrongDirectionRejected, which commits an encap tunnel. - TunnelIpInIpValidationTest declares nothing and holds the tests the CLI rejects, or treats as a no-op, before anything reaches the agent. The encap fixture declares nothing on purpose: asic_production_features lists IP_IN_IP_ENCAP for one ASIC only, but the encap tests pass on others, e.g. Tomahawk5. Fixture-level declarations: - PFC: ConfigPfcTest, DeletePfcTest - COPP: ConfigCoppTest, ConfigCoppTrafficPolicyTest, DeleteCoppTest - INGRESS_MIRRORING, EGRESS_MIRRORING: ConfigInterfaceMirrorTest The tunnel tests get a new suite name, so their result history restarts.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-submission checklist
Summary
Stacked on #2023; only the last commit belongs to this PR.
Adds the first production feature declarations to the CLI integration fixtures. They are inert until
run_test.pyis given--enable-production-features.Fixture split
TunnelIpInIpTestkeeps the helpers and becomes the base of three fixtures:TunnelIpInIpDecapTestIP_IN_IP_DECAPTunnelIpInIpEncapTestDeleteWrongDirectionRejected(it commits an encap tunnel)TunnelIpInIpValidationTestThe tunnel tests get a new suite name, so their result history restarts.
Fixture-level declarations
PFC:ConfigPfcTest,DeletePfcTestCOPP:ConfigCoppTest,ConfigCoppTrafficPolicyTest,DeleteCoppTestINGRESS_MIRRORING,EGRESS_MIRRORING:ConfigInterfaceMirrorTestDeliberately not declared
asic_production_featuresrecords what the agent tests verify per ASIC. A CLI test only needs the agent to accept and program the config, so in our hardware runs some CLI tests pass on ASICs the table does not list the matching feature for. Declaring those features would filter out passing tests, so they are left out:IP_IN_IP_ENCAPConfigPfcTest.RxPfcDurationEnablePFC_RX_DURATIONConfigPfcTest.TxPfcDurationEnablePFC_TX_DURATIONVLAN, LAG, PTP, sFlow, MAC learning and load-balancing fixtures are also left undeclared: the table omits those features for
qumran4dand we have no results to confirm the tests fail there.Open question for reviewers
IP_IN_IP_DECAPhas the same shape: the table omits it for tomahawk6, yet the decap tests pass there in our runs. With this declaration,--enable-production-features tomahawk6filters them out. We kept the declaration to follow the table; if a CLI config test should not be gated this way, or the table should gain the entry, we can drop or adjust it.Test plan
fboss2_integration_testcompiles and links. The run as a whole did not finish, becausefboss/agent/hw/sai/api/bcm/SwitchApi.cppfrom currentmaindoes not compile against the prebuilt SAI our CI uses (SAI_SWITCH_ATTR_EXT_STATIC_FDB_REPLACE_DYNAMICundeclared); this PR does not touch that file. The C++ unit test step did not run for the same reason.--list_production_featureon the built binary reportsIP_IN_IP_DECAPfor theTunnelIpInIpDecapTesttests,PFCfor the PFC fixtures,COPPfor the CoPP fixtures andINGRESS_MIRRORING,EGRESS_MIRRORINGforConfigInterfaceMirrorTest; every other test reports an empty list.Fboss2IntegrationTestRunner._filter_testswas run against the built binary and the checked-inasic_production_featuresJSON: tomahawk5 and g202x keep every test, tomahawk6 and qumran4d dropTunnelIpInIpDecapTest, ebro dropsConfigPfcTest.Review Findings
Pre-publication review of the final diff found no build or correctness issues. One note: the
IP_IN_IP_DECAPdeclaration filters tests that pass on tomahawk6, as described under "Open question for reviewers".