fix(txpool): prevent removed transactions from being included in blocks

[wo-o]

Problem

A race condition exists where BestTransactions iterator creates an independent snapshot via BTreeMap::clone(). When the maintenance job removes transactions from the pool, existing iterators remain unaware, causing removed transactions to be included in blocks.

Production Impact

During load testing with --txpool.lifetime 600, this caused sequencer failure:

2025-11-15T13:30:01.790071Z WARN The database read transaction has been open for too long.
    open_duration=190.664088693s self.txn_id=7039115

2025-11-15T13:30:01.947723Z WARN Attempt to calculate state root for an old block
    might result in OOM target=2846869

The sequencer stopped producing blocks because:

1. Maintenance removed expired transactions from pool
2. Block builder's snapshot still contained the removed transactions
3. Execution attempted to process non-existent transactions
4. Database read stuck
5. State root calculation failed

This is critical for transactions with time-based expiration.

Root Cause

crates/transaction-pool/src/pool/pending.rs:110 - The iterator snapshot is created via BTreeMap::clone(), which creates an independent copy. The pool has notification support for new transactions but lacks removal notifications.

Solution

Add removal notifications following the existing pattern for new transactions:

1. PendingPool broadcasts transaction removals via new removed_transaction_notifier channel
2. BestTransactions subscribes to removal notifications on creation
3. Iterator processes removals before yielding next transaction

[mattsse]

could you elaborate on why

causing removed transactions to be included in blocks.

would be problematic, because atm I believe this would only happen if the pending pool is at capacity, but I assume you need this to satisfy some protocol level rules?

Block builder's snapshot still contained the removed transactions
Execution attempted to process non-existent transactions
Database read stuck

not following the sequence of events here that's causing db/State root issues, maybe this has something to do with some protocol specific rules?
if the blockbuilder adds this tx into the block, how would this cause execution errors?

[mattsse]

lacking some context on why excluding removed txs is critical

but maybe this is similar to OP interop inclusion rules

reth/crates/optimism/txpool/src/transaction.rs

Lines 104 to 115 in 9f3949c

	impl<Cons, Pooled> MaybeInteropTransaction for OpPooledTransaction<Cons, Pooled> {
	fn set_interop_deadline(&self, deadline: u64) {
	self.interop.store(deadline, Ordering::Relaxed);
	}
	fn interop_deadline(&self) -> Option<u64> {
	let interop = self.interop.load(Ordering::Relaxed);
	if interop > NO_INTEROP_TX {
	return Some(interop)
	}
	None
	}

which is checked during block building for example

reth/crates/optimism/payload/src/builder.rs

Line 696 in 9f3949c

let interop = tx.interop_deadline();

reth/crates/optimism/payload/src/builder.rs

Lines 730 to 737 in 9f3949c

	// We skip invalid cross chain txs, they would be removed on the next block update in
	// the maintenance job
	if let Some(interop) = interop &&
	!is_valid_interop(interop, self.config.attributes.timestamp())
	{
	best_txs.mark_invalid(tx.signer(), tx.nonce());
	continue
	}

[mattsse]

not opposed to this, but since this is an internal channel we can unify this by introducing

enum PendingPoolEvent<T> {Added(Tx),Removed(tx)}

[wo-o]

@mattsse Thanks for the feedback. I think my original description was misleading.

The issue is not that removed transactions get included in blocks. The actual problem is that when a transaction is removed from the pool (e.g., due to --txpool.lifetime expiration), the BestTransactions iterator still holds references to it in its snapshot. During block building, this mismatch causes the iterator to get stuck trying to access transactions that no longer exist in the pool's state, leading to long-running DB read transactions and eventually blocking block production.

I've also unified the two notification channels into a single PendingPoolEvent enum as you suggested:
0f2d0af

[mattsse]

leading to long-running DB read transactions and eventually blocking block production.

can you expand on this a bit more, because I don't quite understand how this would cause issues during building, is there something that just endlessly loops then? and how exactly

[mattsse]

@wo-o we had another recent fix for updates in this one c7b6890 unsure if related, but we now have a smol conflict, sorry about this.

I have one more questions, because it's unclear how still behaviour would stall block production because this iter is a full snapshot, basically independent of the pool