Crash loop on startup: SIGILL/stack overflow in ACP ChainedHandler dispatch after agent-client-protocol 2.0.0 bump

[NewtonChutney]

Note: This report was investigated with the assistance of Claude (Anthropic's AI coding assistant) — it reproduced the crash, performed the bisect below, and traced the root cause to the linked upstream PR, all verified by me before filing. Flagging this upfront per the "(for AI issues)" convention on this tracker.

Reproduction steps

1. Have a workspace with an ACP thread open, backed by an external agent server (in my repro, an ACP-bridged agent talking to a local LM Studio instance).
2. Launch Zed (cargo run --bin zed, debug build) so the workspace/session restores that thread.
3. Within roughly 60–150 seconds of launch, Zed crashes.
4. Relaunching reproduces the same crash again (crash loop), since the same thread state gets restored.

This reproduces from a clean checkout of main with no local modifications — no custom settings or extensions needed beyond having that ACP thread persisted.

Zed version and system specs

Zed: v1.14.0 (dev build, cargo run --bin zed)
OS: macOS 26.5.2 (build 25F84)
Architecture: aarch64

Reproduced directly against main at 82aef44308 ("gpui: Add an abstraction for scheduling foreground work for idle periods, and implement it for the web (#61827)") — with agent-client-protocol pinned to 2.0.0 per Cargo.lock at that commit.

Attach Zed log file

Nothing relevant was written to Zed.log — the crash happens below the level the app's own logger can catch (it's an illegal-instruction trap on a background dispatch worker thread, not a Rust panic). The macOS crash reporter's .ips diagnostic report is what actually captured it; the relevant parts are transcribed below.

---

Crash signature

EXCEPTION: EXC_BAD_ACCESS (SIGILL), subtype KERN_PROTECTION_FAILURE
message: "Could not determine thread index for stack guard region"

This message is emitted when Rust's stack-overflow guard-page handler can't identify the overflowing thread, which happens on GCD worker threads (_dispatch_worker_thread2 → pthread_wqthread) — these have a much smaller stack than the main thread.

The faulting thread's top frames are all inside agent_client_protocol::jsonrpc::handlers, with ChainedHandler<ChainedHandler<ChainedHandler<...>>> repeated many times while dispatching a RequestPermissionRequest:

agent_client_protocol::jsonrpc::handlers::RequestHandler::handle_dispatch_from (RequestPermissionRequest)
agent_client_protocol::role::handle_incoming_dispatch
agent_client_protocol::jsonrpc::handlers::RequestHandler::handle_dispatch_from
agent_client_protocol::jsonrpc::handlers::ChainedHandler::handle_dispatch_from
agent_client_protocol::jsonrpc::handlers::ChainedHandler::handle_dispatch_from
agent_client_protocol::jsonrpc::handlers::ChainedHandler::handle_dispatch_from
... (repeats several more times)

I reproduced this three separate times with the identical signature (only the guard-page address differs between runs).

Bisect

I bisected against the agent-client-protocol dependency's version history in Cargo.lock rather than a blind commit range, since the crash is entirely inside that crate:

Commit	agent-client-protocol	Result
d1548bd013 ("repl: Show add-cell controls in empty notebooks", #61329)	1.3.0	Ran cleanly for 4+ minutes with the same workspace state, no crash
984e3bd0ce ("acp: Update agent-client-protocol SDK to 2.0.0", #61570)	2.0.0	Crashed within ~90 seconds, identical signature

Both were verified directly (checked out, built, and run) rather than inferred from the log alone.

Root cause (best guess)

ChainedHandler<H1, H2> in agent-client-protocol 2.0.0 (src/jsonrpc/handlers.rs) is:

pub struct ChainedHandler<H1, H2> {
    handler1: H1,
    handler2: H2,
}

impl<Counterpart: Role, H1, H2> HandleDispatchFrom<Counterpart> for ChainedHandler<H1, H2>
where
    H1: HandleDispatchFrom<Counterpart>,
    H2: HandleDispatchFrom<Counterpart>,
{
    async fn handle_dispatch_from(&mut self, message: Dispatch, connection: ConnectionTo<Counterpart>) -> Result<Handled<Dispatch>, crate::Error> {
        match self.handler1.handle_dispatch_from(message, connection.clone()).await? {
            Handled::Yes => Ok(Handled::Yes),
            Handled::No { message, retry: retry1 } => match self.handler2.handle_dispatch_from(message, connection).await? {
                // ...
            }
        }
    }
}

Nothing here boxes or type-erases H1/H2. Zed's crates/agent_servers/src/acp.rs registers 9 handlers via on_receive_request!(), so the compiler builds a ChainedHandler<ChainedHandler<ChainedHandler<...H9>>> where each nesting level's generated async fn state machine embeds the entire next level's state inline. In an unoptimized (dev) build this produces an enormous unboxed future, and dispatching a request on a worker thread with a smaller stack overflows it. This looks like the same class of bug Zed has hit (and fixed) elsewhere with unbounded/oversized stack usage — e.g. #58325, #51637, #41397, #35813 — usually addressed with stacker-style dynamic stack growth or by boxing the chain.

This was introduced upstream by agentclientprotocol/rust-sdk PR #277 ("refactor(acp)!: clarify routing and handler APIs", part of the 2.0.0 release) and pulled into Zed via #61570. I couldn't find an existing issue or PR addressing it in either repo.

Relevant Zed settings

An ACP-bridged external agent server configured against a local LM Studio instance was open in the restored workspace when this reproduced.

(for AI issues) Model provider details

External agent via Agent Client Protocol (ACP), bridging to a local LM Studio instance.

[NewtonChutney]

I no longer am able to repro this issue though.. might be something related to the state I was in, that triggered it?
nvm triggered it.. I had to use an ACP thread, not a zed agent thread

[NewtonChutney]

Reproduced again on current main at ae394f3d47 (v1.15.0, dev build, cargo run --bin zed), same signature as the original bisect. Attaching the fresh crash report details.

Exception

EXCEPTION: EXC_BAD_ACCESS (SIGILL), subtype KERN_PROTECTION_FAILURE
message: "Could not determine thread index for stack guard region"

Faulting thread: 37 of 39, on GCD queue com.apple.root.default-qos — a background worker thread, not the main thread.

Bottom of the stack (confirms it's a small-stack GCD worker thread)

start_wqthread
_pthread_wqthread
_dispatch_worker_thread2
_dispatch_root_queue_drain
_dispatch_client_callout
gpui_macos::dispatcher::trampoline
<async_task::runnable::Runnable<scheduler::RunnableMeta>>::run
<async_task::raw::RawTask<..., scheduler::RunnableMeta>>::run
<core::pin::Pin<alloc::boxed::Box<dyn core::future::future::Future<Output = ()> + core::marker::Send>> as core::future::future::Future>::poll

Top of the stack (the runaway dispatch chain)

Frame 0:

<agent_client_protocol::jsonrpc::handlers::RequestHandler<agent_client_protocol::role::acp::Agent, agent_client_protocol::role::acp::Agent, agent_client_protocol_schema::v1::client::RequestPermissionRequest, ...> as agent_client_protocol::jsonrpc::HandleDispatchFrom<agent_client_protocol::role::acp::Agent>>::handle_dispatch_from

27 of the 41 frames reference ChainedHandler in their mangled symbol.

Frame size as evidence of the unboxed nesting

Demangled symbol length grows almost linearly from frame 5 through frame 20/21, then drops — each additional chained handler embeds the entire next level's generic type inline:

Frame	Demangled length (chars)
5	3,097
10	10,374
15	16,242
20	33,831
21	33,762
25	18,142
30	17,360
31–41	14–343 (drops to the executor/dispatcher frames above)

This is on agent-client-protocol 2.0.0 (unchanged in Cargo.lock since the original report), with crates/agent_servers/src/acp.rs unmodified since — the conditions from the original bisect are still current.

[NewtonChutney]

Note: This verification was performed with the assistance of Claude (Anthropic) — it set up the [patch.crates-io] override, built the patched binary, and drafted this comment; I ran the actual repro against it and confirmed the result before posting.

Verified the fix in agentclientprotocol/rust-sdk#306 resolves this.

Patched agent-client-protocol locally against the PR's branch at its current head (5b9454a21cf6ddc4b3edee994f115dd604adfd70, pnkfelix/acp-rust-sdk) via [patch.crates-io], keeping everything else on current main (ae394f3d47, v1.15.0):

[patch.crates-io]
agent-client-protocol = { git = "https://github.com/pnkfelix/acp-rust-sdk", rev = "5b9454a21cf6ddc4b3edee994f115dd604adfd70" }

Built cleanly (cargo update -p agent-client-protocol + cargo build --bin zed, no version conflicts — the fork still declares 2.0.0, satisfying Zed's =2.0.0 pin). Ran the exact same repro conditions that reliably crashed before (restored workspace with an open ACP thread against an external agent, waited the full ~150s window): no crash.

PR agentclientprotocol/rust-sdk#306's own diagnosis matches the crash report posted above — ChainedHandler inlining each sub-handler's future into its parent's poll frame, scaling stack usage with handler-chain depth (Zed registers ~10-20 handlers) — and its fix (boxing each link so the future lives on the heap) needs no Zed-side changes, just a version bump past =2.0.0 once it's released. Worth tracking that PR and bumping agent-client-protocol here once it lands.

[yeskunall]

Thanks @NewtonChutney -- I can repro this too, but this issue should only exist on macOS on dev builds ONLY.

[NewtonChutney]

Ohh.. Is there any other way to work around it when the dev build is stuck unable to launch?

I run a fork of zed, https://github.com/NewtonChutney/zed, with support for the vertex provider.. And I'm working on a custom ACP agent harness..