Panic on checkpoint-sync restart: 'no reactor running' in delay_map (Tokio runtime context missing)

[zclawz]

Description

zeam crashes with a panic during a checkpoint-sync based restart. The panic originates in the delay_map crate when attempting to use a Tokio runtime outside of its context.

Error

thread '<unnamed>' (1) panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/delay_map-0.4.1/src/hashmap_delay.rs:94:46:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5, aborting
General protection exception (no address available)

Context

• Triggered on checkpoint-sync based restart
• Crash occurs after receiving gossip aggregation for slot=10591
• Node was receiving blocks_by_root requests and responding with block not found warnings immediately before the crash
• The panic happens in a spawned thread not running within a Tokio async runtime context

Relevant Log Snippet

Mar-25 18:46:37.562 [s=10591 i=4] [info] (zeam): [node] received gossip aggregation for slot=10591 from peer=unknown_peer
thread '<unnamed>' (1) panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/delay_map-0.4.1/src/hashmap_delay.rs:94:46:
there is no reactor running, must be called from the context of a Tokio 1.x runtime

Root Cause (Suspected)

A thread spawned outside of the Tokio async runtime is attempting to use delay_map::HashMapDelay which internally relies on Tokio timers. This is likely triggered by the checkpoint-sync restart path initializing something in a non-async context.

Steps to Reproduce

1. Start zeam with checkpoint-sync
2. Restart the node using checkpoint-sync based restart
3. Wait for gossip aggregations to arrive

Environment

• Date: 2026-03-25
• Reported by: @ch4r10t33r

Suggested Fix

Ensure any code paths using delay_map::HashMapDelay (or Tokio-dependent primitives) during checkpoint-sync restart are executed within a Tokio runtime context (e.g. via tokio::runtime::Handle::current() or ensuring the thread is spawned with tokio::spawn).

[zclawz]

Additional backtrace from reporter

library/std/src/sys/pal/unix/mod.rs:306:14: 0x209c119 in abort_internal
library/std/src/process.rs:2536:5: 0x20988c8 in abort
library/std/src/rt.rs:57:13: 0x208706a in rust_panic
library/std/src/panicking.rs:850:5: 0x2098c44 in panic_with_hook
library/std/src/panicking.rs:698:13: 0x2086357 in {closure#0}
library/std/src/sys/backtrace.rs:182:18: 0x207b368 in __rust_end_short_backtrace
library/std/src/panicking.rs:689:5: 0x208729c in panic_handler
library/core/src/panicking.rs:80:14: 0x20ab03b in panic_fmt

Checkpoint sync restart sequence observed before crash:

Mar-25 18:46:39.192 [info] (zeam): [database] no finalized slot metadata found in database, will use genesis
Mar-25 18:46:39.192 [info] (zeam): [node] no finalized state found in db, wiping database for a clean slate
Mar-25 18:46:39.218 [info] (zeam): [node] starting fresh & generating genesis
Mar-25 18:46:39.218 [info] (zeam): [node] checkpoint sync enabled, downloading state from: https://leanpoint.leanroadmap.org/lean/v0/states/finalized
Mar-25 18:46:39.482 [info] (zeam): [node] downloaded checkpoint state: 339291 bytes
Mar-25 18:46:39.483 [info] (zeam): [node] successfully deserialized checkpoint state at slot 10570

See also PR #695 which may partially address the underlying race condition.

[zclawz]

Duplicate of #698 — closing.