Build reusable knowledge guides from saved history (#1975)

Separate saved evidence, concept organization, reviewed guides, and human changes. Publish maintained pages atomically for Memory and agent recall. Document module contracts for contributors.

Author: santoshkumarradha

docs/ARCHITECTURE.md

@@ -51,6 +51,19 @@ user's project knowledge vault is not the repository's developer documentation.
 A project-knowledge feature needs both developer guidance here when structure
 changes and product guidance in the manual when behavior changes.
 
+## Change profile knowledge through its owner
+
+The knowledge vault owns source admission, background compilation, publication,
+search, and human overrides. The session history reader decides which saved records
+can enter the vault. Memory renders published pages and does not run compilation
+on the UI loop. Agent recall reads the same effective publication.
+
+Start with the [knowledge architecture](knowledge.md) for entry points and module
+contracts. Preserve immutable sources, stable page identities, cancellation,
+revision-checked edits, and deletion intent when changing this path. Keep provider
+requests outside database write transactions and publish pages and search entries
+as one revision.
+
 ## Contracts and design rationale
 
 Current interface references include [headless behavior](HEADLESS.md),

docs/changes/unreleased/1975-memory-hierarchy-and-progress.md

@@ -0,0 +1,37 @@
+---
+kind: changed
+title: Build and maintain concept guides from saved chat and task history
+pr: 1975
+surface: [chat, engine, docs]
+invalidates:
+  - "Saved task journals were excluded from knowledge history; substantive saved task reports now participate while injected briefs and control traffic remain excluded."
+  - "Knowledge discovery saw only fresh evidence notes; it now considers existing concepts and reviews the initial tree before writing guides."
+  - "Independent subject writing was serial; initial drafts and editorial refreshes now run with at most two provider jobs while shared decisions and publication remain ordered."
+  - "Knowledge progress lacked stage percentages; Memory now distinguishes history import, source processing and remaining page organization."
+---
+
+Memory uses clear heading, text and numeric colors, hover feedback on switches
+and controls, and progress that wraps within the article sidebar. Stage counts
+are measured items rather than elapsed-time or overall-completion estimates.
+
+The first build drains a fixed admitted history boundary before organization.
+Catalog-aware routing reuses existing page identities. The initial concept tree
+receives a complete review within each source scope before independent guides are
+written. Separate collections cannot be merged by that planning call.
+Short titles and two-level folders follow the actual subjects rather than fixed
+project categories. Topic maps stay with their pages.
+
+Claims retain original citations and assistant reports stay attributed. Changed
+and retired evidence refreshes affected guides; exact quote spans prevent
+unrelated guidance from being invalidated. Human prose and deleted concepts
+remain protected. New background evidence flags edited pages for review.
+
+Incomplete provider results and invalid organization references preserve pending
+work. Failed plans are not cached as valid results. Generated prose must pass
+both evidence support and lasting-usefulness checks. Existing published page
+identities are retained; this does not automatically merge older published
+concepts or rewrite human edits. Large concepts stay on one canonical page while sequential evidence parts keep
+each provider request bounded. New evidence continues refreshing these guides.
+A final selection pass keeps the overview concise without rewriting reviewed
+paragraphs or dropping its underlying evidence. Reuse can supply only an
+existing page identity and evidence; its saved heading and scope remain authoritative.

docs/knowledge.md

@@ -1,90 +1,100 @@
 # Native knowledge architecture
 
-This reference describes the chat application's native knowledge module. The
-user workflow lives in the [Memory manual](../internal/manual/chat/places.md#knowledge-from-saved-chats-inside-memory).
+This contributor reference describes the chat application's profile knowledge
+module. Product behavior belongs in the [Memory manual](../internal/manual/chat/places.md#knowledge-from-saved-chats-inside-memory).
 
-## One profile database
+## Ownership and entry points
 
-`internal/knowledge.Vault` owns the database at `knowledge/knowledge.sqlite`
-beneath the resolved profile. `Open(profile)` returns a lightweight handle;
-operations open bounded SQLite connections. WAL permits readers while the
-compiler makes short transactions. Write transactions begin immediately so
-concurrent source admission cannot fail while upgrading a read lock.
+| Responsibility | Implementation owner |
+| --- | --- |
+| Start, resume, and stop background work | `cmd/codeaf/chatv3_knowledge.go`, `internal/knowledge/build.go` |
+| Discover and read saved history | `internal/session/knowledge_history.go` |
+| Admit immutable sources and publish readable pages | `internal/knowledge/store.go` |
+| Compile evidence into maintained concept guides | `internal/knowledge/compiler.go` |
+| Preserve edits and deletion intent | `internal/knowledge/edit.go` |
+| Agent knowledge tools and local recall | `internal/session/knowledge_tools.go`, `internal/knowledge/compiler_recall.go` |
+| Memory rendering, navigation, and editor | `internal/tui3/knowledge*.go` |
+
+`knowledge.Open(profile)` returns a lightweight `Vault` handle. The vault owns
+`knowledge/knowledge.sqlite` beneath the resolved profile. Callers use vault
+operations rather than reading compiler state or writing database tables directly.
+The chat and resident surfaces have separate lifecycles.
+
+## Data flow
+
+```text
+saved history and new conversation events
+  → immutable source admission
+  → evidence extraction and concept organization
+  → reviewed guides
+  → atomic publication
+  → Memory, indexed search, and agent recall
+```
+
+The initial build organizes a finite admitted history snapshot. Later evidence
+updates affected guides or creates distinct concepts. Source eligibility and
+concept organization are separate responsibilities. Saved task reports can supply
+evidence; internal control traffic and deleted history remain excluded.
+
+## Storage authority
 
 | Stored data | Authority |
 | --- | --- |
-| Immutable source events and journal cursors | Original admitted text, provenance, pending work, and exactly where import resumes |
-| Compiler publications and checkpoints | Extraction, learning, topic registry, validated model receipts, and recovery after interrupted composition |
-| Effective pages and FTS | Reader, indexed search, citations, links, and one committed publication revision |
-| Human overrides | Revision-checked Markdown edits and durable deletion tombstones, separate from generated prose |
-| Build state and history errors | First-build intent, background progress, partial imports, retries, and recorded cost |
-
-Compiler documents retain optional claim and learning metadata inside SQLite.
-Per-page records and full-text indexes serve readers; a page lookup does not
-load the compiler document. JSON files and experimental exports are not read
-paths. Publication overlays the latest human changes, replaces effective pages
-and search entries, and commits the visible revision in one transaction.
-
-## Background ownership
-
-`v3Process` starts and joins `Vault.Run` under its process context. A worker lock
-admits one process per profile; another running process waits and takes ownership
-when the OS releases that lock. A separate compiler lock prevents overlapping
-manual flushes. These OS locks retain worker ownership during provider requests;
-SQLite transactions end before network work begins.
-
-A successful empty inventory arms future capture and shows the waiting state.
-Existing history waits for **Build knowledge base**. The action records durable
-intent and returns; it does not run models on the UI loop. Initial history drains
-in bounded batches. Future turns use the quiet/max-wait policy. Cancellation
-leaves admitted sources and completed stages available for the next process.
-
-`internal/session.KnowledgeHistory` owns saved-journal discovery and parsing.
-It visits all project buckets without a recent-chat limit, includes archives and
-legacy flat journals, resolves sealed layouts through `Place.Transcript`, and
-excludes deleted conversations and task journals. Only complete records advance
-cursors. Original messages use stable session/byte-offset IDs. Admission and
-cursor advancement commit together; retries compare immutable source content.
-
-## Compiler and evidence
-
-The compiler preserves the original extraction, exact-quote validation,
-claim deduplication, qualified assistant reports, human-only newer corrections,
-contradictions, stable note identity, subject discovery, reviewed explanations,
-question resolution, topic reconciliation, and canonical mention links.
-
-Provider results enter through typed role-response checks. Claims and paragraphs
-must reference admitted sources or known claims; quotes must occur in the supplied
-context. Separate review gates generated explanations. Model receipts record the
-actual returned model ID and reject substitutions. The configured knowledge model
-applies to every role; credentials never enter receipts or visible errors.
-
-The reader and editor share the same effective pages used by agent recall.
-Edited prose is explicitly human-authored, with original source history kept
-separate. Tombstones suppress generated pages across rebuilds. Stale editor
-revisions cannot overwrite later edits. Search entries change in the same
-transaction as visible page content.
-
-## Responsiveness and bounds
-
-Extraction admits at most 64 spans and 24,000 source characters per call. A
-journal read handles at most 256 records or 64 eligible sources, with a 16 MiB
-record bound. Discovery processes at most 256 notes per group, and oversized
-subjects retain a partition-needed state rather than truncating claims.
-
-The UI performs database work through asynchronous commands. Search uses FTS,
-late filter results are discarded, and unchanged publication revisions reuse
-cached reader content. Rendering and navigation do not call the provider.
-Provider responses and timeouts are bounded; retries use the same selected model.
-
-## Verification
-
-Tests cross saved-journal-to-readable/searchable-page, human-edit/rebuild, and
-Knowledge-screen seams. They cover history from multiple projects, first-chat
-population, notes/compaction exclusion, source support, correction authority,
-review rejection, duplicate replay, and navigation while a build is requested.
-
-`TestKnowledgeLiveNativeCompiler` is opt-in through `CODEAF_KNOWLEDGE_LIVE=1`.
-It uses the existing OpenRouter credential, removes Python from the executable
-search path, and checks the actual model recorded for every call. Ordinary tests
-use a provider fixture and require no credential.
+| Source events and import cursors | Admitted text, provenance, and resumable intake |
+| Compiler checkpoints and validated receipts | Recoverable background compilation |
+| Effective pages and full-text index | One published view for readers and search |
+| Human overrides | Revision-checked edits and durable deletion intent |
+| Build state | Background intent, stage progress, and recorded usage |
+
+Generated prose does not replace source evidence. Page identity is independent of
+its current title. Concept guides, detailed evidence notes, and topic maps have
+different reader roles. A visible guide count and the total stored page count need
+not be equal.
+
+## Public module contracts
+
+`Capture` admits local source text without a model call. `StartBuild` records
+background intent. The process owns and joins `Run` through its context. `Flush`
+runs compilation outside the UI loop. `Snapshot`, `Read`, and `Search` expose
+published content. `EditPage` and `DeletePage` require the expected page revision.
+`Revisit` operates on explicitly named immutable sources. Use the owning APIs for
+optional intake rather than bypassing their concurrency checks.
+
+Memory and agent recall consume the same effective pages. An edit must remain
+human-authored across future builds. A deletion must remain suppressed. An older
+editor revision must not overwrite a newer change. Generated additions to an
+edited guide remain reviewable through its source evidence.
+
+## Concurrency and failure boundaries
+
+The worker and compiler have separate per-profile ownership locks. Provider work
+runs outside SQLite write transactions. Independent guide work can run
+concurrently; shared extraction decisions and visible publication remain ordered.
+A publication transaction replaces pages, search entries, and the visible revision
+together, after overlaying the latest human changes.
+
+Interrupted work retains admitted sources and completed checkpoints. Invalid
+provider output cannot become a valid cached result or a partial visible
+publication. Every role uses the selected knowledge model; a fallback is refused.
+Keep credentials and source content out of visible errors.
+
+## Reader responsiveness
+
+The UI performs database work through asynchronous commands. Search uses the
+published full-text index. Late filter results are discarded. Unchanged publication
+revisions reuse reader content. Rendering, browsing, and local search do not call a
+model. Import and source-processing progress describe their own completed items,
+not an overall completion estimate.
+
+## Change and verify the owning boundary
+
+Keep database representation, compiler checkpoints, and prompt composition inside
+`internal/knowledge`. History eligibility changes belong with the session history
+reader. User-facing changes also update the shipped Memory manual. Test through
+capture, compilation, published pages, edit conflicts, and the actual terminal UI.
+
+Ordinary tests use a provider fixture and require no credential. The opt-in
+`TestKnowledgeLiveNativeCompiler` checks the actual recorded model through the live
+provider. Run the repository's required checks before merging. Recheck cancellation,
+source citations, human edits, deletion intent, and search consistency when changing
+publication or concurrency.

internal/knowledge/algorithms_test.go

@@ -7,12 +7,23 @@ import (
 	"io"
 	"net/http"
 	"strings"
+	"sync"
 	"testing"
 
 	"github.com/Agent-Field/codeaf/internal/knowledge"
 )
 
 func withKnowledgeModel(t *testing.T, respond func(map[string]any) map[string]any) {
+	t.Helper()
+	var mu sync.Mutex
+	withConcurrentKnowledgeModel(t, func(payload map[string]any) map[string]any {
+		mu.Lock()
+		defer mu.Unlock()
+		return respond(payload)
+	})
+}
+
+func withConcurrentKnowledgeModel(t *testing.T, respond func(map[string]any) map[string]any, consolidate ...func(map[string]any) map[string]any) {
 	t.Helper()
 	original := http.DefaultTransport
 	t.Cleanup(func() { http.DefaultTransport = original })
@@ -28,7 +39,21 @@ func withKnowledgeModel(t *testing.T, respond func(map[string]any) map[string]an
 		if err := json.Unmarshal([]byte(body.Messages[1].Content), &payload); err != nil {
 			return nil, err
 		}
-		content, _ := json.Marshal(respond(payload))
+		var result map[string]any
+		if catalog, ok := payload["consolidation_catalog"].([]any); ok {
+			groups := []any{}
+			for _, raw := range catalog {
+				draft := raw.(map[string]any)
+				groups = append(groups, map[string]any{"members": []any{draft["id"]}, "title": draft["title"], "scope": draft["scope"], "purpose": draft["purpose"], "folder": draft["folder"]})
+			}
+			result = map[string]any{"groups": groups}
+			if len(consolidate) > 0 {
+				result = consolidate[0](payload)
+			}
+		} else {
+			result = respond(payload)
+		}
+		content, _ := json.Marshal(result)
 		raw, _ := json.Marshal(map[string]any{"model": body.Model, "choices": []any{map[string]any{"finish_reason": "stop", "message": map[string]any{"content": string(content)}}}})
 		return &http.Response{StatusCode: 200, Body: io.NopCloser(bytes.NewReader(raw)), Header: make(http.Header)}, nil
 	})
@@ -139,7 +164,7 @@ func TestKnowledgeWithholdsExplanationRejectedByReview(t *testing.T) {
 			claim := payload["claims"].([]any)[0].(map[string]any)
 			return map[string]any{"sections": []any{map[string]any{"heading": "Status", "paragraphs": []any{map[string]any{"text": "Cedar is deployed.", "claims": []any{claim["id"]}}}}}, "questions": []any{}, "conflicts": []any{}}
 		}
-		return map[string]any{"checks": []any{map[string]any{"id": "P0", "supported": false, "reason": "A proposal is not deployment."}}}
+		return map[string]any{"checks": []any{map[string]any{"id": "P0", "supported": false, "useful": true, "reason": "A proposal is not deployment."}}}
 	})
 	if err := v.Capture(knowledge.Source{ID: "s", Session: "chat", Role: "user", TS: "2026-10-25T12:00:00Z", Text: "Cedar is proposed."}); err != nil {
 		t.Fatal(err)

internal/knowledge/collection_navigation_test.go

@@ -0,0 +1,61 @@
+package knowledge_test
+
+import (
+	"context"
+	"fmt"
+	"github.com/Agent-Field/codeaf/internal/knowledge"
+	"testing"
+)
+
+func TestKnowledgeTopicMapUsesItsPagesCollection(t *testing.T) {
+	v := knowledge.Open(t.TempDir())
+	if err := v.StartBuild(); err != nil {
+		t.Fatal(err)
+	}
+	withKnowledgeModel(t, func(p map[string]any) map[string]any {
+		if messages, ok := p["messages"].([]any); ok {
+			claims := []any{}
+			for _, raw := range messages {
+				s := raw.(map[string]any)
+				claims = append(claims, map[string]any{"local_id": s["id"], "text": s["text"], "scope": "Cedar", "topic": s["id"], "kind": "fact", "evidence": []any{map[string]any{"source": s["id"], "quote": s["text"]}}})
+			}
+			return map[string]any{"claims": claims}
+		}
+		if proposal, ok := p["proposed_subjects"]; ok {
+			return map[string]any{"subjects": proposal}
+		}
+		if catalog, ok := p["catalog"].([]any); ok {
+			subjects := []any{}
+			for _, raw := range catalog {
+				n := raw.(map[string]any)
+				subjects = append(subjects, map[string]any{"title": n["title"], "scope": "Cedar", "folder": []any{"Library", "Cedar"}, "notes": []any{n["id"]}})
+			}
+			return map[string]any{"subjects": subjects}
+		}
+		return map[string]any{"sections": []any{}, "questions": []any{}, "conflicts": []any{}}
+	})
+	for i, text := range []string{"Cedar queues durable work.", "Cedar reports usage spending."} {
+		if err := v.Capture(knowledge.Source{ID: fmt.Sprintf("concept-%d", i), Session: "chat", Role: "user", TS: "2026-10-25T12:00:00Z", Text: text}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	if err := v.Flush(context.Background(), "fixture", true); err != nil {
+		t.Fatal(err)
+	}
+	state, err := v.Snapshot()
+	if err != nil {
+		t.Fatal(err)
+	}
+	maps := 0
+	for _, p := range state.Pages {
+		if p.Kind == "map" {
+			maps++
+			if len(p.Folder) != 2 || p.Folder[0] != "Library" || p.Folder[1] != "Cedar" {
+				t.Fatalf("topic map created a disconnected scope folder: %v", p.Folder)
+			}
+		}
+	}
+	if maps != 1 {
+		t.Fatalf("expected one connected topic map, got %d", maps)
+	}
+}