fix: increase block production timeouts to account for event loop lag (#7420)

**Motivation**

Based on observations from mainnet nodes, it seems like we reject
builder blocks in some cases either due to not being sent within cutoff
time or due to timeout on the api call but looking at the response times
these have been timely. The reason why those were rejected is either we
started the block production race too late into the slot, which is
mostly due to the fact that we take too much time to produce the common
block body or the timeout was handled by the node with a delay, both of
these cases are likely caused by event loop lag either due to GC or
processing something else.

See
[discord](https://discord.com/channels/593655374469660673/1331991458152058991/1335576180815958088)
for details.

**Description**

Increase block production timeouts to account for event loop lag

Author: nflaig

packages/beacon-node/src/api/impl/validator/index.ts

@@ -102,10 +102,10 @@ export const SYNC_TOLERANCE_EPOCHS = 1;
  * Post this time, race execution and builder to pick whatever resolves first
  *
  * Empirically the builder block resolves in ~1.5+ seconds, and execution should resolve <1 sec.
- * So lowering the cutoff to 2 sec from 3 seconds to publish faster for successful proposal
- * as proposals post 4 seconds into the slot seems to be not being included
+ * So lowering the cutoff to 2.5 sec from 3 seconds to publish faster for successful proposal
+ * as proposals post 4 seconds into the slot will likely be orphaned due to proposer boost reorg.
  */
-const BLOCK_PRODUCTION_RACE_CUTOFF_MS = 2_000;
+const BLOCK_PRODUCTION_RACE_CUTOFF_MS = 2_500;
 /** Overall timeout for execution and block production apis */
 const BLOCK_PRODUCTION_RACE_TIMEOUT_MS = 12_000;
 

packages/beacon-node/src/execution/builder/http.ts

@@ -53,11 +53,17 @@ export class NoBidReceived extends Error {
   }
 }
 
+/**
+ * Additional duration to account for potential event loop lag which causes
+ * builder blocks to be rejected even though the response was sent in time.
+ */
+const EVENT_LOOP_LAG_BUFFER = 250;
+
 /**
  * Duration given to the builder to provide a `SignedBuilderBid` before the deadline
  * is reached, aborting the external builder flow in favor of the local build process.
  */
-const BUILDER_PROPOSAL_DELAY_TOLERANCE = 1000;
+const BUILDER_PROPOSAL_DELAY_TOLERANCE = 1000 + EVENT_LOOP_LAG_BUFFER;
 
 export class ExecutionBuilderHttp implements IExecutionBuilder {
   readonly api: BuilderApi;