teardown() sequence removes interface routes before flushing DNAT rules, causing packet leaks to default gateway

[FabienLahoudere]

When stopping containerized services or proxies (e.g., via podman stop / systemctl stop), we observe a race condition where incoming packets on mapped host ports are DNAT'd and forwarded to the host's default gateway on the physical egress interface (eth0), rather than being dropped or rejected locally.

In strict enterprise network environments using SDN switches with active dataplane endpoint learning, receiving packets on the physical network with internal container destination IPs (e.g., 10.89.0.x or RFC 1918 private subnets) triggers incorrect endpoint learning and routing table corruption, leading to network outages.

Root Cause Analysis in Netavark
Inspecting src/network/bridge.rs reveals two flaws in the teardown() function lifecycle:

1. Inverted Teardown Order (FIFO instead of LIFO)

teardown() calls remove_link() before self.teardown_firewall():

Rust

// 1. Link and container routes are deleted FIRST
let complete_teardown = match remove_link(...) { ... };

// 2. Firewall / DNAT / Port-forwarding rules are deleted LAST
match self.teardown_firewall(complete_teardown, bridge_name) { ... }

Consequence:

remove_link() destroys the virtual veth/bridge interface and removes the host route to the internal container subnet (10.89.0.0/24).

During the time window before teardown_firewall() is reached, DNAT rules remain active in the kernel.

Incoming client packets (e.g., from 192.0.2.100) arriving on the host port are translated to the internal container IP (10.89.0.2).

Since the container interface and route no longer exist, the Linux kernel falls back to the host's default route (eth0), spitting the translated packet out to the physical external network.

2. Error Handling & Dangling Rules

If remove_link() returns an error, complete_teardown is set to false. This prevents teardown_firewall() from running properly, leaving DNAT rules active indefinitely without a backing network interface.

Solution

A DNAT rule must never outlive the route to its target. We would like to discuss the best approach to resolve this upstream:

• Re-ordering Teardown (LIFO): Can the teardown process be split or re-ordered so that container-specific DNAT/port-forwarding rules are removed BEFORE remove_link() destroys the internal routes?
• Conntrack Flush: Should Netavark explicitly flush conntrack entries associated with the container IP during teardown to avoid existing connections lingering through the transition?
• Fallback Handling: What is the recommended way to handle errors in remove_link() to ensure firewall cleanup is never skipped?

Thanks for your feedback

[Luap99]

Re-ordering Teardown (LIFO): Can the teardown process be split or re-ordered so that container-specific DNAT/port-forwarding rules are removed BEFORE remove_link() destroys the internal routes?

The code checks if we have any connected interface to know that is can do the fully network teardown which is needed for the firewall. But I guess we could reorder the check to check for only one connected interface, then do firewall and then remove the last interface from the bridge so that should be possible.

Conntrack Flush: Should Netavark explicitly flush conntrack entries associated with the container IP during teardown to avoid existing connections lingering through the transition?

conntrack works is stalled on external library work last I checked but yes we should remove stale conntrack entries on startup and stop
#1362
I think we only observed problems with udp so far but I guess it could just as well be extended to tcp as well if needed.

Fallback Handling: What is the recommended way to handle errors in remove_link() to ensure firewall cleanup is never skipped?

Well the error is handled as best as we can already, the firewall teardown is not skipped fully, the per container rules such as ports should still be removed. Only the full network subnet rules should not removed as we did not remove the bridge either when that failed and other contianer could still be running on it

[frispete]

Allow me to add some remarks:

Since the container interface and route no longer exist, the Linux kernel falls back to the host's default route (eth0), spitting the translated packet out to the physical external network.

..with the unfortunate side effect that these packages then include the eth.src from the podman host, ip.src and eth.dst are from the minion (which is completely nonsense already!) and the ip.dst of the internal container address. No wonder, that SDN switches are confused at this point.

I think we only observed problems with udp so far but I guess it could just as well be extended to tcp as well if needed.

We observed this behaviour with salt, which is definitely using tcp. The best outcome of this fix would be, if the kernel would plainly reject these requests as long as the container isn't handling any network traffic. Specifically during, and even more importantly, after teardown!

If remove_link() returns an error, complete_teardown is set to false. This prevents teardown_firewall() from running properly, leaving DNAT rules active indefinitely without a backing network interface.

From observed behaviour, we can conclude, that the teardown sequence fails in such a way, that the DNAT kept going, way beyond the small time window of the wrong teardown sequence ordering, implying, that

let complete_teardown = match remove_link(

fails more often, than probably expected.

[Luap99]

fails more often, than probably expected.

I doubt that, there is no reason for this to fail randomly and errors will be logged and returned to podman so they should show up somewhere in your logs, check journald. And as mentioned if we actually talk about the port forward DNAT rules they are always removed regardless of of remove_link failing or not.