Test flake in `chaos_tests::port_settings::test_port_settings_txn_sweep`

[bnaecker]

Seen on #106, here. The full error is below, but the buildomat output has much more detail:

3911	2025-08-29T00:39:18.570Z	Error: operation failed: desired settings: PortSettings {
3912	2025-08-29T00:39:18.570Z	    links: {},
3913	2025-08-29T00:39:18.570Z	}
3914	2025-08-29T00:39:18.570Z	current settings: PortSettings {
3915	2025-08-29T00:39:18.570Z	    links: {
3916	2025-08-29T00:39:18.570Z	        "0": LinkSettings {
3917	2025-08-29T00:39:18.570Z	            addrs: [],
3918	2025-08-29T00:39:18.570Z	            params: LinkCreate {
3919	2025-08-29T00:39:18.570Z	                autoneg: true,
3920	2025-08-29T00:39:18.570Z	                fec: Some(
3921	2025-08-29T00:39:18.570Z	                    None,
3922	2025-08-29T00:39:18.570Z	                ),
3923	2025-08-29T00:39:18.570Z	                kr: false,
3924	2025-08-29T00:39:18.571Z	                lane: Some(
3925	2025-08-29T00:39:18.571Z	                    LinkId(
3926	2025-08-29T00:39:18.571Z	                        0,
3927	2025-08-29T00:39:18.571Z	                    ),
3928	2025-08-29T00:39:18.571Z	                ),
3929	2025-08-29T00:39:18.571Z	                speed: Speed100G,
3930	2025-08-29T00:39:18.571Z	                tx_eq: None,
3931	2025-08-29T00:39:18.571Z	            },
3932	2025-08-29T00:39:18.571Z	        },
3933	2025-08-29T00:39:18.571Z	    },
3934	2025-08-29T00:39:18.571Z	}

[bnaecker]

I hit this a second time on the same PR. It looks like we have a 5s timeout waiting for some of these random port settings to take effect.

[bnaecker]

Another flake on #113 here.

[rcgoodfellow]

Thanks Ben. I noticed that in the cases you reported this happens when applying port settings that have no links. I updated the test to use port settings with no links every 10 iterations instead of random settings and I can now readily reproduce this locally. Looking into what's going on.

[rcgoodfellow]

So link deletion is now asynchronous with respect to port_settings_apply. That means the implementation of this function can no longer detect a failure to delete a link. The expectation when port_settings_apply was written was that anything created, modified or destroyed by the function would be done so synchronously so the function could detect failures and have atomic semantics modulo rollback failures. Now that this premise is invalid, we're seeing failures.

[rcgoodfellow]

I do like the approach of having the in-memory soft state model with an internal reconciler working to drive the hardware toward that state. But if we're going there, everything should be built on that model. That means port_settings_apply would only be making updates to that soft state model (this is not currently the case, there are a number of actual hardware changes it makes such as table updates for things like addressing) and those updates would carry the transactional semantics the function had previously that went all the way down to the hardware. This is probably better than the prior approach because it does not have to deal with the possibility of rollback failure, and the underlying machinery is just constantly driving the hardware toward a well defined state.

[rcgoodfellow]

Disabling this test in #182 for now