mem::smart-search crashes the whole iii worker — one request unregisters all 271 functions for ~5s

[i-sannikov]

mem::smart-search crashes the whole iii worker — one request unregisters all 271 functions for ~5s

Summary

A single request to POST /agentmemory/smart-search (or one call to the memory_smart_search MCP tool) takes down the entire iii worker, not just that function. Immediately after the request, every endpoint — including /agentmemory/health and /agentmemory/sessions — fails with Function ... not found. The worker reconnects on its own after roughly 5 seconds and everything works again. The node process itself does not restart (uptime keeps climbing).

The blast radius is what makes this worth reporting: the failing function is one endpoint, but the observable symptom is "agentmemory is broken" — the viewer hangs on load, session lists time out, and the dashboard shows its empty state ("agentmemory is running but hasn't seen any sessions yet") even though 339 sessions are stored and readable.

Environment

@agentmemory/agentmemory	0.9.28 (global npm install)
iii engine	0.11.2 (pinned)
Node	v26.5.0
OS	Darwin 25.5.0, arm64 (macOS, Apple Silicon)
Store	file-based, ~/.agentmemory/data, ~503 MB
Scale	339 sessions, 160 observation scopes, 300 lessons, 108 summaries, 3273 semantic memories, 404 crystals, 921 insights, graph 9474 nodes / 8663 edges
Embeddings	EMBEDDING_PROVIDER=openai → ollama bge-m3 @ localhost:11434/v1, 1024 dims
LLM	deepseek deepseek-v4-flash

Reproduction

# 1. baseline — everything healthy
curl -s http://127.0.0.1:3111/agentmemory/health   | jq '.health.workers[0].function_count'   # 271
curl -s http://127.0.0.1:3111/agentmemory/sessions | jq '.sessions | length'                  # 339

# 2. one smart-search request
curl -s -X POST http://127.0.0.1:3111/agentmemory/smart-search \
  -H 'Content-Type: application/json' -d '{"query":"x","limit":1}'
# => HTTP 500 {"error":"Invocation stopped","error_id":"db326f1f8608"}

# 3. immediately after — the whole worker is gone
curl -s http://127.0.0.1:3111/agentmemory/health
# => HTTP 500 {"error":"Function middleware::api-auth not found"}
curl -s http://127.0.0.1:3111/agentmemory/sessions
# => HTTP 500 {"error":"Function api::sessions not found"}

# 4. wait ~5s — fully recovered, no restart needed
sleep 5
curl -s http://127.0.0.1:3111/agentmemory/health   | jq '.health.workers[0].function_count'   # 271
curl -s http://127.0.0.1:3111/agentmemory/sessions | jq '.sessions | length'                  # 339

Measured timeline:

Moment	/agentmemory/health	/agentmemory/sessions
before	200, function_count: 271	200, 339 sessions, 0.0s
right after one smart-search	500 Function middleware::api-auth not found	500 Function api::sessions not found
+5s	200, function_count: 271	200, 339 sessions, 0.0s
+15s / +30s	200	200

Reproduced on every attempt, with the crash occurring ~0.5–0.6s into the request.

Narrowing it down

Each case below was run only after waiting for function_count to be back at 271, so the results are independent:

Request body	Result	Worker survived?
{"query":"x","limit":1}	500 Invocation stopped (0.6s)	no
{"query":"x","limit":1,"includeLessons":false}	500 Invocation stopped (0.5s)	no
{"expandIds":["obs_..."]}	200 {"mode":"expanded",...}	yes
{}	400 query or expandIds is required	yes

So the expandIds branch and the validation path are both fine, and includeLessons: false does not help — which points at the searchFn(data.query, overFetchLimit) call in the query branch of mem::smart-search rather than at recallLessons.

Notably, plain POST /agentmemory/search with the same query returns 200 with results and leaves the worker alive, so the keyword/BM25 path is healthy. That isolates the fault to the hybrid/vector portion reached only via smart-search.

What is not the cause

Ruled out during diagnosis, in case it saves you time:

• Not the data. memory_diagnose reports 0 failures (13 pass / 9 warn), all summaries/semantic/crystals/insights consistent, and /agentmemory/sessions returns all 339 sessions in 0.0s when the worker is up.
• Not the store size. 339 sessions serialize instantly; there is no slow path here, only a crash.
• Not embeddings. ollama answers on localhost:11434/v1, bge-m3:latest is present, and POST /v1/embeddings returns a 1024-dim vector in 0.2s.
• Not the LLM. POST /chat/completions against the configured deepseek endpoint returns 200 in 1.0s with a valid key.
• Not a stale install. Fresh npm install -g @agentmemory/agentmemory@latest, and the crash persists across a clean restart where the node process and the engine both start together.

Impact

Because the crash unregisters all functions, unrelated requests in flight or issued within that window fail in confusing ways:

• The viewer on :3113 hangs on load indefinitely.
• GET /agentmemory/sessions issued inside the crash window never completes and eventually returns 504 after exactly 180s (default_timeout: 180000 for the iii-http worker in iii-config.yaml).
• The dashboard renders its first-run empty state, telling the user to run the demo, while hundreds of real sessions sit in the store.
• The memory_smart_search MCP tool silently returns {"results": []} instead of surfacing the error, so agents get "no results" rather than "the backend just crashed" — and each call knocks the worker out for the next ~5 seconds.

Suggestions

1. Contain the failure: an exception inside one registered function should not deregister the worker's whole function table.
2. Surface it: have the MCP tool propagate the 500 instead of returning an empty result set.
3. The root exception is presumably visible in the engine log against error_id (e.g. db326f1f8608, db33065be0c0) — happy to attach that output, or run any instrumented build you point me at.