mem::graph-extract hangs to its own 180s invocation timeout on every batch, eventually stalling state::get and crashing the worker

[deemaxx]

Setup

• agentmemory: @agentmemory/agentmemory@0.9.27 at time of incident (since upgraded to 0.9.28 during investigation — hang persisted after upgrade based on log/behavior review, not independently re-verified with a fresh 7-day soak)
• Runtime: macOS (darwin arm64), launched via LaunchAgent (com.agentmemory.server.plist)
• LLM provider: local Ollama via OpenAI-compatible endpoint (OPENAI_BASE_URL=http://localhost:11434/v1, OPENAI_MODEL=qwen2.5-coder:7b)
• AGENTMEMORY_LLM_TIMEOUT_MS=120000
• GRAPH_EXTRACTION_ENABLED=true
• Hourly cron (com.agentmemory.graph-build.plist, StartInterval: 3600) calling POST /agentmemory/graph/build with {"batchSize":10}

Symptom

Starting ~2026-07-23T19:57Z, every hourly graph/build cron invocation returned an empty response body (curl -s -m 3300 ... timed out or returned nothing) — this repeated roughly every hour for 7 straight days with zero successful runs, versus clean {"batches":N,"edges":N,"nodes":N,"sessions":N,"success":true} results before that point.

Server-side log (~/Library/Logs/agentmemory-server.log) shows the actual failure, repeated across many sessions and restarts:

[agentmemory] warn graph-build batch failed {"sessionId":"<id>","batchIndex":N,"error":"Invocation timeout after 180000ms: mem::graph-extract"}

This fires for many batchIndex values per session, across multiple distinct sessions, across at least 9 separate worker restarts recorded in the log. Confirmed live: POST /agentmemory/graph/build from a plain curl -v hangs and returns zero bytes even after 20s (well under both the 120s LLM timeout and the 180s function-invocation timeout, so this isn't just "slow LLM").

At least once, the accumulated hang escalated further and stalled the graph snapshot read itself:

[agentmemory] warn Graph snapshot read failed {"error":"Invocation timeout after 180000ms: state::get"}

This appears to correlate with full worker crashes/restarts (KeepAlive.SuccessfulExit=false in the plist relaunches the process, which then repeats the same pattern).

Ollama itself was confirmed healthy and responsive throughout (curl .../v1/chat/completions returns 200 with a real completion in ~1-2s), ruling out "LLM provider unreachable" as the cause.

Impact

Because the hourly cron's curl -s silently swallowed the failed/empty response with no alerting, this ran undetected for 7 days. Worse: this appears to have contributed to (though is not confirmed as the sole cause of) a broader persistence gap where no new session data was durably written for the same window — investigated separately, root-caused to an unrelated relative-path config issue (iii-config.yaml's ./data/state_store.db + missing launchd WorkingDirectory), but the graph-extract hang's periodic worker crashes were part of what obscured that second issue for so long.

What I could confirm vs. couldn't

• Confirmed: mem::graph-extract invocations reliably fail via 180s invocation timeout, not an LLM-side error/4xx.
• Confirmed: this happens across restarts, across sessions, consistently over multiple days — not a one-off.
• Not confirmed: the exact internal cause of the hang (I don't have visibility into mem::graph-extract's implementation internals/what it's blocked on — no stack trace or deadlock diagnostic surfaces in the log).
• Not confirmed: whether this reproduces with a non-Ollama provider (Anthropic/OpenAI direct) — my setup only used local Ollama.

Suggested improvements (independent of root cause)

1. graph-build-cron.sh (or whatever calls /agentmemory/graph/build) should treat an empty/timeout response as a hard failure and alert loudly, not silently no-op.
2. Repeated mem::graph-extract timeouts should probably trip a circuit breaker / back off rather than retrying every hour indefinitely against what's evidently a stuck code path.
3. A hung mem::graph-extract batch shouldn't be able to stall unrelated state::get calls — that coupling turned a contained failure (graph feature degraded) into a full worker crash (session capture degraded too).

Happy to provide the full log excerpt or iii-config.yaml if useful.