memory: add forget/demote (+ observation decay) so stale memories can be superseded

[marcmantei]

Problem

The agent's long-term memory has no way to forget or demote a stale memory. There is only memory_save; the memory HTTP API (/api/agents/memories*) is read-only (list/search/graph). So once a wrong or obsolete belief is saved at high importance, it persists — and repetitive "observation" memories keep themselves warm via constant access, dominating recall and drowning out newer corrections.

Evidence (operator memory-refresh, 2026-06-19)

The supervisor (main) had accumulated ~50 "worker-healer alert storm / false-positive since May 31" observation memories, all importance: 1.0, with access_count 79–86 and last_accessed/updated_at = today. These made it (a) decline heavy tasks on an obsolete "worker-timeout death-spiral" premise and (b) reflexively skip/confabulate on healer alerts (it invented "8 contradictory alerts" / "docker binary missing", both false).

We loaded 4 authoritative CORRECTION … SUPERSEDES … memories via /api/portal/send. They persisted and are retrievable — but they're outranked in generic recall by the ~50 storm memories, which stay warm through repeated access. Recency doesn't help; the stale cluster never decays. There is no tool or API to demote it, and editing agent.db's forgotten flag directly wouldn't affect the Lance-backed search without an index rebuild (and risks desync).

Proposed

1. memory_forget agent tool (and/or memory_supersede): let the agent mark a memory forgotten / lower its importance / supersede it by id. Must update the search index (Lance FTS/vector) so forgotten memories drop out of memory_recall and /agents/memories/search, not just the SQLite row.
2. Decay / TTL for low-value repetitive observations: importance decay over time and/or de-duplication of near-identical recurring observations (the alert-storm cluster is ~50 variations of one observation). A 10-day repeated "false positive" note should not sit at importance 1.0 forever.
3. Operator memory-mutation API (optional): authenticated PATCH/DELETE on /agents/memories/{id} (set forgotten, adjust importance) so corrections like feat: add dedicated voice model routing and attachment transcription spacedriveapp/spacebot#98 can be applied deterministically instead of hoping memory_save out-ranks the noise.

Acceptance

• The agent can forget/supersede a memory by id or description, and it no longer surfaces in memory_recall / search (index updated).
• Repetitive observation memories decay in importance (or de-dupe), so a stale narrative can't dominate recall indefinitely.
• An authoritative correction (newer memory_save) reliably out-ranks the obsolete cluster it supersedes.

Context

Follow-up to shipyard spacedriveapp#98 (supervisor memory refresh). The corrections are loaded; this issue is what's needed to make such refreshes actually stick. The healer false-positive root causes themselves are already fixed in shipyard (spacedriveapp#115, spacedriveapp#117, spacedriveapp#122).

[shipyard-ci]

🚀 Working on this issue. Generating PRD, then implementing.

Routed as task_execution (85%) → task_execution_flow · Triggered by @marcmantei · View AI traces

[shipyard-ci]

PRD — memory: add forget/demote (+ observation decay) so stale memories can be superseded

Product Requirements Document (PRD)

Issue: #20 — Memory Forget/Demote & Observation Decay

---

Objective

Enable the agent to actively manage stale memories by forgetting, demoting, or superseding them, and implement decay mechanisms so repetitive observations don't dominate recall indefinitely. Ensure corrections reliably out-rank obsolete beliefs in memory search results.

---

Background

Problem: The agent's long-term memory system is write-once and read-only:

• Only memory_save writes; the /api/agents/memories* API lacks mutation endpoints
• Stale, high-importance memories persist indefinitely
• Repetitive "observation" memories stay warm via repeated access, drowning out newer corrections
• Example: ~50 false-positive "worker-healer alert storm" memories (importance 1.0, access_count 79–86, last_accessed today) prevented the agent from accepting corrections and caused confabulation

Impact: Authoritative corrections loaded via /api/portal/send exist but are outranked in generic recall by the stale cluster, making memory refreshes ineffective.

---

Requirements

R1: Memory Forget/Supersede Tool

• Implement memory_forget agent tool (and/or memory_supersede):
  • Accept memory ID or description as input
  • Mark memory as forgotten and lower its importance
  • Optionally accept a superseding memory ID to link corrections
  • Must update the search index (Lance FTS/vector) so forgotten memories drop out of memory_recall and /api/agents/memories/search
  • Do not rely on SQLite forgotten flag alone; ensure Lance index is rebuilt or invalidated atomically

R2: Observation Decay / De-duplication

• Implement importance decay over time for low-value repetitive observations:
  • Define decay schedule (e.g., time-based TTL or access-count threshold)
  • Detect near-identical recurring observations and de-duplicate or cluster them
  • Ensure a 10-day-old repeated false-positive does not remain at importance 1.0
  • Example: The 50-memory alert-storm cluster should be consolidated and decayed

R3: Operator Memory-Mutation API (Optional)

• Extend /api/agents/memories/{id} endpoints:
  • Support authenticated PATCH to set forgotten, adjust importance, add superseded_by link
  • Support authenticated DELETE (soft-delete via forgotten flag)
  • Ensure mutations also update Lance index atomically
  • Use this for deterministic correction application (alternative to hoping memory_save out-ranks noise)

R4: Correction Ranking

• Ensure newer authoritative corrections reliably out-rank obsolete clusters in generic recall
  • Implement a ranking strategy (recency, importance, source trust level, supersession links)
  • Verify that a correction memory with lower access_count but higher importance/recency surfaces above a stale cluster

---

Acceptance Criteria

• Forgetting Works: Agent can forget/supersede a memory by ID or description; it no longer appears in memory_recall output or /api/agents/memories/search results
• Index Updated: Lance FTS/vector index is atomically updated when a memory is forgotten (not just SQLite row)
• Decay Applied: Repetitive observation memories decay in importance over time or are de-duplicated; a stale narrative cannot dominate recall indefinitely
• Corrections Stick: An authoritative correction (newer memory_save or API mutation) reliably surfaces above the obsolete cluster it supersedes in search results
• No Silent Desync: All mutations (forget, supersede, decay) are applied atomically to both SQLite and Lance; no index-rebuild side effects without operator awareness

---

Out of Scope

• Root-cause fixes for healer false positives (already addressed in shipyard fix(channel): prevent bot spamming from retrigger cascades spacedriveapp/spacebot#115, feat(security): add auth middleware, SSRF protection, shell hardening, and encrypted secrets spacedriveapp/spacebot#117, fix: replace .expect()/.unwrap() with proper error propagation in production code spacedriveapp/spacebot#122)
• Automatic memory garbage collection or pruning (decay is configurable, not mandatory)
• Frontend UI for memory mutation (API implementation sufficient for this issue)
• Versioning or audit trail for memory mutations

---

Open Questions

1. Lance Index Rebuild: How should Lance FTS/vector index be updated atomically when a memory is forgotten?

   • Requires verification: Is there an incremental delete API, or must the index be rebuilt?
   • Reference: docs/environment.md — confirm Lance version and index strategy

2. Decay Schedule: What is the decay function?

   • Linear, exponential, or step-based?
   • TTL threshold (e.g., 10 days) or access-count threshold?
   • Should decay be configurable per agent or global?

3. De-duplication Strategy: How are "near-identical" observations defined?

   • Vector similarity threshold (cosine distance)?
   • Substring matching?
   • Require operator approval or automatic?

4. Ranking Strategy: What ranking factors determine if a correction surfaces above stale observations?

   • Importance weight vs. recency weight?
   • Should superseded_by link boost the superseding memory?
   • Should agent-internal corrections (memory_save) have different weight than operator-sent corrections?

5. API Authentication: For the optional operator mutation API, what auth mechanism applies?

   • Existing /api/portal/send auth model or new token scope?

---

Context Notes

• Related Issue: feat: add dedicated voice model routing and attachment transcription spacedriveapp/spacebot#98 (supervisor memory refresh) — this issue enables refreshes to "actually stick"
• Example Impact: The alert-storm cluster made the supervisor reflexively skip/confabulate on healer alerts (invented "8 contradictory alerts", "docker binary missing" — both false)
• Current Limitation: Editing agent.db's forgotten flag directly risks desync without index rebuild

---

PRD generated. Implementation starting automatically.
Routed as task_execution (85%) → task_execution_flow · View AI traces