diff --git a/LIVE_KBS_MVP.md b/LIVE_KBS_MVP.md new file mode 100644 index 0000000..1f66d84 --- /dev/null +++ b/LIVE_KBS_MVP.md @@ -0,0 +1,38 @@ +# Live KBs MVP + +## Summary + +- Add `LivePack` as a deterministic mutable overlay on top of an immutable mounted base pack. +- Keep the existing flattened postings index, Cortex, and `query()` pipeline unchanged in phase 1. +- Persist with full-snapshot `serialize()` only; no mutable postings map, patch-pack format, or mutation log yet. + +## Implementation + +- Add a new `packages/core/src/live.ts` module with `createLivePack()` and `LivePack`. +- Make live mutations require stable doc ids; the initial `docs` array follows the same rule. +- `addDocument()` inserts or replaces, `updateDocument()` merges partial fields onto the last known full doc and shadows any base copy, and `removeDocument()` tombstones the id while deleting any overlay copy. +- Keep internal state as a read-only `base`, an overlay `Map` of docs, a `Set` of tombstones, a rebuilt `delta` pack, and only the deterministic build knobs that can be recomputed from docs alone. +- Rebuild only the overlay pack after each mutation with the existing `buildPack()` and `mountPack()` primitives. +- Query base and delta separately with the existing lexical/graph options, collapse duplicate ids with delta winning, and sort final hits by `score desc`, `source/docId asc`, then `blockId asc`. +- Return the same `Hit[]` shape as `query()` so downstream code can swap in live packs without adapter changes. +- Keep live querying lexical/graph-only in v1; semantic build options are rejected for live packs until we add a mutation-time embedding story. +- Make `serialize()` materialize the merged live state in stable id order and return a normal `.knolo` snapshot. +- Export `LivePack`, `createLivePack()`, and `LivePackOptions` from `packages/core/src/index.ts`. +- Update the root README and `packages/core/README.md` to document `LivePack`, clarify that Cortex remains a separate append-only memory layer, and keep `knolo dev` as the watch/rebuild workflow instead of adding `build --watch` now. +- Leave `packages/core/src/indexer.ts`, `packages/core/src/query.ts`, and the CLI command behavior unchanged for this phase. + +## Test Plan + +- Add a `node:test` suite under `packages/core/test` for the live pack lifecycle. +- Cover adding a doc, updating a base doc, removing a base doc, re-adding a removed id, query merging, and `serialize()` round-tripping through `mountPack()`. +- Verify repeated `serialize()` calls on the same live state are byte-identical. +- Verify deterministic merge order for equal-score hits and delta-over-base precedence. +- Verify missing ids and updates to unknown ids fail fast. + +## Assumptions + +- Phase 1 is lexical/graph-only; semantic live updates are deferred to a later `consolidate()` or embedding-aware follow-up. +- Anonymous blocks in an existing base pack stay queryable but are read-only for live mutations. +- Rebuilding the overlay pack on every mutation is the intended MVP tradeoff for incremental evidence and docs-scale writes. +- Patch packs, mutation logs, source-watcher APIs, and `build --watch` stay out of scope for this phase. +- No Rust/ICP changes are needed because the persisted output remains a standard `.knolo` pack. diff --git a/README.md b/README.md index 268aa52..a67be10 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Knolo is a **local-first knowledge base engine** built around deterministic retr It provides: -* `@knolo/core` — pack format + deterministic retrieval engine +* `@knolo/core` — pack format + deterministic retrieval engine, LivePack overlay, and Cortex memory layer * `@knolo/cli` — build workflows for `.knolo` artifacts * `create-knolo-app` — instant Next.js starter with playground * `@knolo/langchain` — LangChain-style retriever adapter @@ -64,6 +64,8 @@ npm run knolo:build npm run dev ``` +For pack workflows, `knolo dev` is the watch/rebuild loop for configured sources. We are keeping that workflow instead of adding a separate `build --watch` command in this phase. + Open: ``` @@ -89,16 +91,42 @@ Knolo is: You build `.knolo` packs once. You mount them anywhere — Node, web, React Native, offline. +When you need a deterministic mutable overlay on top of a mounted pack, use `LivePack`. Retrieval is lexical-first and deterministic by default. Hybrid semantic reranking is optional and **never replaces lexical grounding**. +# 🧪 Live KBs MVP + +`LivePack` is the phase-1 mutable overlay for mounted packs. The base pack stays immutable, live docs are keyed by stable ids, and `serialize()` returns a standard `.knolo` snapshot. + +For the rollout plan, implementation notes, and test matrix, see [`LIVE_KBS_MVP.md`](./LIVE_KBS_MVP.md). + +```ts +import { createLivePack, mountPack } from '@knolo/core'; + +const base = await mountPack({ src: './dist/knowledge.knolo' }); +const live = await createLivePack(base, [ + { id: 'notes.alpha', text: 'alpha note', namespace: 'notes' }, +]); + +await live.updateDocument({ id: 'notes.alpha', text: 'alpha note v2' }); +await live.removeDocument('notes.alpha'); +await live.addDocument({ id: 'notes.alpha', text: 'alpha note restored' }); + +const snapshot = await live.serialize(); +const rebuilt = await mountPack({ src: snapshot }); +``` + +`LivePack` stays lexical/graph-only in v1. Cortex remains a separate append-only memory layer. + --- # 🧠 Knolo Cortex Knolo Cortex adds a local-first overlay memory layer on top of `@knolo/core`. +It is separate from `LivePack`: Cortex is append-only memory, while LivePack is a mutable document overlay for mounted packs. It gives you: diff --git a/packages/core/README.md b/packages/core/README.md index de025f3..e8b6d0c 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -28,8 +28,10 @@ It is: * A versioned binary pack format * A deterministic lexical retrieval engine +* A deterministic `LivePack` overlay for mounted packs * An optional semantic rerank layer * A portable knowledge runtime +* A separate append-only Cortex memory layer You build once. You mount anywhere — Node, browser, React Native, serverless, offline. @@ -147,6 +149,51 @@ Properties: * Namespace-aware * Evaluation-friendly scoring +For iterative pack builds, use `knolo dev` as the watch/rebuild workflow. We are keeping that flow instead of introducing `build --watch` in this phase. + +--- + +## 4️⃣ LivePack Overlay + +`LivePack` is a deterministic mutable overlay on top of a mounted base pack. + +It is phase-1 lexical/graph-only. Stable doc ids are required for the initial `docs` array and for every live mutation, and semantic live updates are rejected until the embedding story exists. + +Construction accepts `LivePackOptions` for graph settings such as `maxEdgesPerDoc`, but semantic live options stay disabled in v1. + +It is designed for document-style live updates: + +* `addDocument()` inserts or replaces a live doc by stable id +* `updateDocument()` merges partial fields onto the last known full doc and shadows any base copy +* `removeDocument()` tombstones a doc id and hides the base copy +* `query()` returns the same `Hit[]` shape as `query(pack, ...)` +* `serialize()` materializes the merged live state as a normal `.knolo` snapshot +* repeated `serialize()` calls on the same state are byte-identical + +Live querying in v1 stays lexical/graph-only. +Semantic build or query options are rejected until live embeddings are added. + +```ts +import { createLivePack, mountPack, query } from '@knolo/core'; + +const base = await mountPack({ src: './dist/knowledge.knolo' }); +const live = await createLivePack(base, [ + { id: 'notes.alpha', text: 'alpha note', namespace: 'notes' }, +]); + +await live.addDocument({ id: 'notes.beta', text: 'beta note' }); +await live.updateDocument({ id: 'notes.alpha', text: 'alpha note v2' }); +await live.removeDocument('notes.beta'); +await live.addDocument({ id: 'notes.beta', text: 'beta note restored' }); + +const hits = live.query('alpha note', { topK: 5 }); +const snapshot = await live.serialize(); +const rebuilt = await mountPack({ src: snapshot }); +const roundTripHits = query(rebuilt, 'beta note', { topK: 5 }); +``` + +For the phase-1 rollout notes and test matrix, see [`../../LIVE_KBS_MVP.md`](../../LIVE_KBS_MVP.md). + --- # 🔀 Optional: Hybrid Semantic Rerank diff --git a/packages/core/package.json b/packages/core/package.json index 0716c6a..39bb00b 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -1,6 +1,6 @@ { "name": "@knolo/core", - "version": "3.2.2", + "version": "3.2.4", "type": "module", "description": "Local-first knowledge packs for small LLMs.", "keywords": [ @@ -35,8 +35,8 @@ "prepublishOnly": "npm run build", "smoke": "node scripts/smoke.mjs", "test": "npm run build && node scripts/check-runtime-no-node.mjs && node --test test/*.test.mjs && node scripts/test.mjs", - "format": "prettier --write src/agent.ts src/pack.ts src/pack.runtime.ts src/pack.node.ts src/node.ts src/builder.ts src/index.ts scripts/test.mjs scripts/check-runtime-no-node.mjs ../../README.md README.md", - "format:check": "prettier --check src/agent.ts src/pack.ts src/pack.runtime.ts src/pack.node.ts src/node.ts src/builder.ts src/index.ts scripts/test.mjs scripts/check-runtime-no-node.mjs ../../README.md README.md", + "format": "prettier --write src/agent.ts src/pack.ts src/pack.runtime.ts src/pack.node.ts src/node.ts src/builder.ts src/live.ts src/index.ts scripts/test.mjs scripts/check-runtime-no-node.mjs ../../README.md README.md", + "format:check": "prettier --check src/agent.ts src/pack.ts src/pack.runtime.ts src/pack.node.ts src/node.ts src/builder.ts src/live.ts src/index.ts scripts/test.mjs scripts/check-runtime-no-node.mjs ../../README.md README.md", "check:runtime-no-node": "node scripts/check-runtime-no-node.mjs" }, "devDependencies": { diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 521658e..c12fc4a 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -8,6 +8,7 @@ export { } from './query.js'; export { makeContextPatch } from './patch.js'; export { buildPack } from './builder.js'; +export { LivePack, createLivePack } from './live.js'; export { quantizeEmbeddingInt8L2Norm, encodeScaleF16, @@ -49,6 +50,7 @@ export { expandQueryWithGraph } from './graph/query_expand.js'; export * from './memory/index.js'; export type { MountOptions, PackMeta, Pack } from './pack.runtime.js'; export type { QueryOptions, Hit } from './query.js'; +export type { LivePackOptions } from './live.js'; export type { EmbeddingProvider, SemanticSidecar, SemanticQueryOptions, RetrievalEvidence } from './semantic/types.js'; export type { ContextPatch } from './patch.js'; export type { BuildInputDoc, BuildPackOptions } from './builder.js'; diff --git a/packages/core/src/live.ts b/packages/core/src/live.ts new file mode 100644 index 0000000..39a2d65 --- /dev/null +++ b/packages/core/src/live.ts @@ -0,0 +1,478 @@ +import { buildPack, type BuildInputDoc } from './builder.js'; +import { buildClaimGraph } from './graph/build_claim_graph.js'; +import type { Pack } from './pack.runtime.js'; +import { mountPack } from './pack.runtime.js'; +import { + query as queryPack, + validateQueryOptions, + type Hit, + type QueryOptions, +} from './query.js'; + +export type LivePackOptions = { + graph?: { + enabled?: boolean; + maxEdgesPerDoc?: number; + }; +}; + +type LiveDoc = BuildInputDoc & { id: string }; + +type BaseDocEntry = { + index: number; + id?: string; + text: string; + heading?: string; + namespace?: string; +}; + +type NormalizedLivePackOptions = { + graph: { + enabled: boolean; + maxEdgesPerDoc?: number; + }; +}; + +export class LivePack { + public readonly base: Readonly; + + private readonly graph: NormalizedLivePackOptions['graph']; + private readonly baseEntries: BaseDocEntry[]; + private readonly baseDocsById: Map; + private overlay = new Map(); + private tombstones = new Set(); + private delta: Pack; + private mutationQueue: Promise = Promise.resolve(); + + constructor(base: Pack, opts: LivePackOptions = {}) { + this.base = base; + this.graph = normalizeLiveGraphOptions(base, opts); + this.baseEntries = extractBaseEntries(base); + this.baseDocsById = indexBaseEntries(this.baseEntries); + this.delta = createEmptyDeltaPack(this.graph.enabled); + } + + public async addDocument(doc: LiveDoc): Promise { + return this.enqueueMutation(async () => { + const nextDoc = normalizeLiveDocument(doc); + const nextOverlay = new Map(this.overlay); + const nextTombstones = new Set(this.tombstones); + + nextOverlay.set(nextDoc.id, nextDoc); + nextTombstones.delete(nextDoc.id); + + const nextDelta = await buildOverlayPack(nextOverlay, this.graph); + this.overlay = nextOverlay; + this.tombstones = nextTombstones; + this.delta = nextDelta; + }); + } + + public async updateDocument(doc: LiveDocumentPatch): Promise { + return this.enqueueMutation(async () => { + const patch = normalizeLivePatch(doc); + const current = this.getMutableDoc(patch.id); + if (!current) { + throw new Error( + `LivePack.updateDocument(...): unknown id "${patch.id}". Use addDocument() to insert new live docs.` + ); + } + + const nextDoc = mergeLiveDoc(current, patch); + const nextOverlay = new Map(this.overlay); + const nextTombstones = new Set(this.tombstones); + + nextOverlay.set(nextDoc.id, nextDoc); + nextTombstones.delete(nextDoc.id); + + const nextDelta = await buildOverlayPack(nextOverlay, this.graph); + this.overlay = nextOverlay; + this.tombstones = nextTombstones; + this.delta = nextDelta; + }); + } + + public async removeDocument(id: string): Promise { + return this.enqueueMutation(async () => { + const normalizedId = normalizeLiveId(id, 'LivePack.removeDocument(...)'); + const current = this.getMutableDoc(normalizedId); + if (!current) { + throw new Error( + `LivePack.removeDocument(...): unknown id "${normalizedId}".` + ); + } + + const nextOverlay = new Map(this.overlay); + const nextTombstones = new Set(this.tombstones); + + nextOverlay.delete(normalizedId); + nextTombstones.add(normalizedId); + + const nextDelta = await buildOverlayPack(nextOverlay, this.graph); + this.overlay = nextOverlay; + this.tombstones = nextTombstones; + this.delta = nextDelta; + }); + } + + public query(q: string, opts: QueryOptions = {}): Hit[] { + validateLiveQueryOptions(opts); + + const topK = opts.topK ?? 10; + const poolTopK = Math.max(25, topK * 5); + const queryOpts = sanitizeLiveQueryOptions(opts, poolTopK); + + const baseHits = queryPack(this.base, q, queryOpts); + const deltaHits = queryPack(this.delta, q, queryOpts); + const hiddenBaseIds = this.getShadowedBaseIds(); + + const merged: Hit[] = []; + for (const hit of baseHits) { + const source = typeof hit.source === 'string' ? hit.source : undefined; + if (source && hiddenBaseIds.has(source)) continue; + merged.push(hit); + } + merged.push(...deltaHits); + + merged.sort(compareHits); + return merged.slice(0, topK); + } + + public async serialize(): Promise { + await this.mutationQueue; + const docs = this.collectMergedDocs(); + const buildOpts: Parameters[1] = this.graph.enabled + ? { + graph: { + enabled: true, + ...(this.graph.maxEdgesPerDoc !== undefined + ? { maxEdgesPerDoc: this.graph.maxEdgesPerDoc } + : {}), + }, + ...(this.base.meta.agents ? { agents: this.base.meta.agents } : {}), + } + : { + graph: { enabled: false }, + ...(this.base.meta.agents ? { agents: this.base.meta.agents } : {}), + }; + + return await buildPack(docs, buildOpts); + } + + private async enqueueMutation(task: () => Promise): Promise { + const run = this.mutationQueue.then(() => task(), () => task()); + this.mutationQueue = run.then( + () => undefined, + () => undefined + ); + return run.then(() => this); + } + + private getMutableDoc(id: string): LiveDoc | undefined { + const overlay = this.overlay.get(id); + if (overlay) return overlay; + if (this.tombstones.has(id)) return undefined; + + const base = this.baseDocsById.get(id); + if (!base) return undefined; + return baseEntryToDoc(base) as LiveDoc; + } + + private getShadowedBaseIds(): Set { + const hidden = new Set(this.tombstones); + for (const id of this.overlay.keys()) hidden.add(id); + return hidden; + } + + private collectMergedDocs(): BuildInputDoc[] { + const hidden = this.getShadowedBaseIds(); + const named = new Map(); + const anonymous: BuildInputDoc[] = []; + + for (const entry of this.baseEntries) { + if (entry.id === undefined) { + anonymous.push(baseEntryToDoc(entry)); + continue; + } + if (hidden.has(entry.id)) continue; + named.set(entry.id, baseEntryToDoc(entry)); + } + + for (const [id, doc] of this.overlay) { + named.set(id, cloneLiveDoc(doc)); + } + + const sortedNamed = [...named.entries()] + .sort(([left], [right]) => compareIds(left, right)) + .map(([, doc]) => doc); + + return [...sortedNamed, ...anonymous]; + } +} + +export async function createLivePack( + base: Pack, + docs: LiveDoc[] = [], + opts: LivePackOptions = {} +): Promise { + const live = new LivePack(base, opts); + for (const doc of docs) { + await live.addDocument(doc); + } + return live; +} + +type LiveDocumentPatch = { + id: string; + text?: string; + heading?: string; + namespace?: string; +}; + +function normalizeLiveGraphOptions( + base: Pack, + opts: LivePackOptions +): NormalizedLivePackOptions['graph'] { + const graphRequested = opts.graph?.enabled; + const inferredEnabled = + graphRequested ?? + (opts.graph?.maxEdgesPerDoc !== undefined ? true : Boolean(base.meta.claimGraph)); + + if (!inferredEnabled) { + return { enabled: false }; + } + + return { + enabled: true, + ...(opts.graph?.maxEdgesPerDoc !== undefined + ? { maxEdgesPerDoc: opts.graph.maxEdgesPerDoc } + : {}), + }; +} + +function extractBaseEntries(base: Pack): BaseDocEntry[] { + return base.blocks.map((text, index) => ({ + index, + text, + heading: base.headings?.[index] ?? undefined, + namespace: base.namespaces?.[index] ?? undefined, + id: normalizeBaseDocId(base.docIds?.[index]), + })); +} + +function indexBaseEntries(entries: BaseDocEntry[]): Map { + const index = new Map(); + for (const entry of entries) { + if (entry.id === undefined) continue; + if (index.has(entry.id)) { + throw new Error( + `LivePack requires stable doc ids. Duplicate base doc id "${entry.id}" is not supported.` + ); + } + index.set(entry.id, entry); + } + return index; +} + +function normalizeBaseDocId(id: string | null | undefined): string | undefined { + if (id === undefined || id === null) return undefined; + const normalized = normalizeLiveId(id, 'base pack'); + return normalized; +} + +function normalizeLiveDocument(doc: LiveDoc): LiveDoc { + if (!doc || typeof doc !== 'object') { + throw new Error('LivePack expects document objects with stable ids.'); + } + + const id = normalizeLiveId(doc.id, 'LivePack document'); + const text = normalizeLiveText(doc.text, `LivePack document "${id}"`); + const heading = validateOptionalString(doc.heading, 'heading', id); + const namespace = validateOptionalString(doc.namespace, 'namespace', id); + + return { + id, + text, + ...(heading !== undefined ? { heading } : {}), + ...(namespace !== undefined ? { namespace } : {}), + }; +} + +function normalizeLivePatch(patch: LiveDocumentPatch): LiveDocumentPatch { + if (!patch || typeof patch !== 'object') { + throw new Error('LivePack.updateDocument(...) expects an object with an id.'); + } + + const id = normalizeLiveId(patch.id, 'LivePack.updateDocument(...)'); + const out: LiveDocumentPatch = { id }; + + if (patch.text !== undefined) { + out.text = normalizeLiveText( + patch.text, + `LivePack.updateDocument("${id}")` + ); + } + if (patch.heading !== undefined) { + out.heading = validateOptionalString(patch.heading, 'heading', id); + } + if (patch.namespace !== undefined) { + out.namespace = validateOptionalString(patch.namespace, 'namespace', id); + } + return out; +} + +function mergeLiveDoc(current: LiveDoc, patch: LiveDocumentPatch): LiveDoc { + const next: LiveDoc = { + id: current.id, + text: current.text, + ...(current.heading !== undefined ? { heading: current.heading } : {}), + ...(current.namespace !== undefined ? { namespace: current.namespace } : {}), + }; + + if (patch.text !== undefined) next.text = patch.text; + if (patch.heading !== undefined) next.heading = patch.heading; + if (patch.namespace !== undefined) next.namespace = patch.namespace; + + return cloneLiveDoc(next); +} + +function cloneLiveDoc(doc: LiveDoc): LiveDoc { + return { + id: doc.id, + text: doc.text, + ...(doc.heading !== undefined ? { heading: doc.heading } : {}), + ...(doc.namespace !== undefined ? { namespace: doc.namespace } : {}), + }; +} + +function baseEntryToDoc(entry: BaseDocEntry): BuildInputDoc { + return { + id: entry.id, + text: entry.text, + ...(entry.heading !== undefined ? { heading: entry.heading } : {}), + ...(entry.namespace !== undefined ? { namespace: entry.namespace } : {}), + }; +} + +function validateLiveQueryOptions(opts: QueryOptions): void { + if (opts.semantic) { + const semanticEntries = Object.entries(opts.semantic).filter( + ([key, value]) => key !== 'enabled' && value !== undefined + ); + if (opts.semantic.enabled === true || semanticEntries.length > 0) { + throw new Error( + 'LivePack.query(...): semantic query options are not supported in v1.' + ); + } + } + validateQueryOptions({ ...opts, semantic: undefined }); +} + +function sanitizeLiveQueryOptions( + opts: QueryOptions, + topK: number +): QueryOptions { + return { + ...opts, + topK, + semantic: undefined, + }; +} + +function createEmptyDeltaPack(graphEnabled: boolean): Pack { + const claimGraph = graphEnabled ? buildClaimGraph([]) : undefined; + return { + meta: { + version: 3, + stats: { + docs: 0, + blocks: 0, + terms: 0, + avgBlockLen: 1, + }, + }, + lexicon: new Map(), + postings: new Uint32Array(0), + blocks: [], + headings: [], + docIds: [], + namespaces: [], + blockTokenLens: [], + ...(claimGraph ? { claimGraph } : {}), + }; +} + +async function buildOverlayPack( + overlay: Map, + graph: NormalizedLivePackOptions['graph'] +): Promise { + const docs = [...overlay.values()].sort((a, b) => compareIds(a.id, b.id)); + const bytes = await buildPack( + docs, + graph.enabled + ? { + graph: { + enabled: true, + ...(graph.maxEdgesPerDoc !== undefined + ? { maxEdgesPerDoc: graph.maxEdgesPerDoc } + : {}), + }, + } + : { + graph: { enabled: false }, + } + ); + return await mountPack({ src: bytes }); +} + +function compareHits(left: Hit, right: Hit): number { + const scoreDiff = right.score - left.score; + if (scoreDiff !== 0) return scoreDiff; + + const leftSource = typeof left.source === 'string' ? left.source : '\uffff'; + const rightSource = typeof right.source === 'string' ? right.source : '\uffff'; + const sourceDiff = compareStrings(leftSource, rightSource); + if (sourceDiff !== 0) return sourceDiff; + + return left.blockId - right.blockId; +} + +function compareIds(left: string, right: string): number { + return compareStrings(left, right); +} + +function compareStrings(left: string, right: string): number { + if (left === right) return 0; + return left < right ? -1 : 1; +} + +function normalizeLiveId(id: unknown, context: string): string { + if (typeof id !== 'string') { + throw new Error(`${context}: id must be a non-empty string.`); + } + if (!id.trim()) { + throw new Error(`${context}: id must be a non-empty string.`); + } + return id; +} + +function normalizeLiveText(text: unknown, context: string): string { + if (typeof text !== 'string' || !text.trim()) { + throw new Error(`${context}: text must be a non-empty string.`); + } + return text; +} + +function validateOptionalString( + value: unknown, + field: 'heading' | 'namespace', + id: string +): string | undefined { + if (value === undefined) return undefined; + if (typeof value !== 'string') { + throw new Error( + `LivePack document "${id}": ${field} must be a string when provided.` + ); + } + return value; +} diff --git a/packages/core/test/live-pack.test.mjs b/packages/core/test/live-pack.test.mjs new file mode 100644 index 0000000..b3268c9 --- /dev/null +++ b/packages/core/test/live-pack.test.mjs @@ -0,0 +1,178 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { + buildPack, + createLivePack, + mountPack, + query, +} from '../dist/index.js'; + +function stableHits(hits) { + return hits.map(({ source, text, namespace }) => ({ + source, + text, + namespace, + })); +} + +function buildDocsSnapshot(pack, docId) { + const index = pack.docIds?.findIndex((id) => id === docId) ?? -1; + if (index < 0) { + throw new Error(`Doc id not found in mounted pack: ${docId}`); + } + return { + text: pack.blocks[index], + heading: pack.headings?.[index] ?? null, + namespace: pack.namespaces?.[index] ?? null, + }; +} + +test('LivePack lifecycle is deterministic and round-trips through mountPack', async () => { + const base = await mountPack({ + src: await buildPack([ + { + id: 'b', + heading: 'Base B', + namespace: 'base', + text: 'alpha same', + }, + { + id: 'shared', + heading: 'Base Shared', + namespace: 'base', + text: 'alpha same', + }, + { + id: 'base-update', + heading: 'Base Heading', + namespace: 'base', + text: 'base note', + }, + { + id: 'remove-me', + heading: 'Remove Heading', + namespace: 'base', + text: 'obsolete unique', + }, + ]), + }); + + const live = await createLivePack(base, [ + { + id: 'a', + heading: 'Live A', + namespace: 'live', + text: 'newterm aaabbc', + }, + { + id: 'z', + heading: 'Live Z', + namespace: 'live', + text: 'newterm aabaad', + }, + { + id: 'shared', + heading: 'Live Shared', + namespace: 'live', + text: 'alpha same', + }, + ]); + + const newtermHits = live.query('newterm', { topK: 5 }); + assert.deepEqual(newtermHits.map((hit) => hit.source), ['a', 'z']); + + const alphaHits = live.query('alpha', { topK: 5 }); + assert.equal(alphaHits.some((hit) => hit.source === 'b'), true); + assert.equal(alphaHits.find((hit) => hit.source === 'shared')?.namespace, 'live'); + + await live.addDocument({ + id: 'added', + heading: 'Added', + namespace: 'live', + text: 'fresh term gamma', + }); + assert.deepEqual(stableHits(live.query('newterm', { topK: 5 })), [ + { source: 'a', text: 'newterm aaabbc', namespace: 'live' }, + { source: 'z', text: 'newterm aabaad', namespace: 'live' }, + ]); + assert.deepEqual(stableHits(live.query('fresh', { topK: 5 })), [ + { source: 'added', text: 'fresh term gamma', namespace: 'live' }, + ]); + + await live.updateDocument({ + id: 'base-update', + text: 'updated note', + }); + assert.deepEqual(stableHits(live.query('updated', { topK: 5 })), [ + { source: 'base-update', text: 'updated note', namespace: 'base' }, + ]); + + await live.removeDocument('remove-me'); + assert.equal(live.query('obsolete', { topK: 5 }).length, 0); + + await live.addDocument({ + id: 'remove-me', + heading: 'Readded Heading', + namespace: 'live', + text: 'restored unique', + }); + assert.deepEqual(stableHits(live.query('restored', { topK: 5 })), [ + { source: 'remove-me', text: 'restored unique', namespace: 'live' }, + ]); + + const bytes1 = await live.serialize(); + const bytes2 = await live.serialize(); + assert.equal(Buffer.compare(Buffer.from(bytes1), Buffer.from(bytes2)), 0); + + const roundTrip = await mountPack({ src: bytes1 }); + const roundTripOpts = { topK: 5, queryExpansion: { enabled: false } }; + assert.deepEqual(stableHits(query(roundTrip, 'newterm', roundTripOpts)), [ + { source: 'a', text: 'newterm aaabbc', namespace: 'live' }, + { source: 'z', text: 'newterm aabaad', namespace: 'live' }, + ]); + assert.deepEqual(stableHits(query(roundTrip, 'fresh', roundTripOpts)), [ + { source: 'added', text: 'fresh term gamma', namespace: 'live' }, + ]); + assert.deepEqual(stableHits(query(roundTrip, 'updated', roundTripOpts)), [ + { source: 'base-update', text: 'updated note', namespace: 'base' }, + ]); + assert.deepEqual(stableHits(query(roundTrip, 'restored', roundTripOpts)), [ + { source: 'remove-me', text: 'restored unique', namespace: 'live' }, + ]); + + assert.deepEqual(buildDocsSnapshot(roundTrip, 'base-update'), { + text: 'updated note', + heading: 'Base Heading', + namespace: 'base', + }); + assert.deepEqual(buildDocsSnapshot(roundTrip, 'remove-me'), { + text: 'restored unique', + heading: 'Readded Heading', + namespace: 'live', + }); + assert.equal(roundTrip.docIds?.filter((id) => id === 'shared').length, 1); + assert.equal(roundTrip.namespaces?.[roundTrip.docIds?.findIndex((id) => id === 'shared') ?? -1], 'live'); +}); + +test('LivePack validates stable ids and unknown updates fail fast', async () => { + const base = await mountPack({ + src: await buildPack([ + { + id: 'known', + text: 'known doc', + }, + ]), + }); + + await assert.rejects( + createLivePack(base, [{ text: 'missing id' }]), + /id/i + ); + + const live = await createLivePack(base); + await assert.rejects( + live.updateDocument({ id: 'missing', text: 'nope' }), + /unknown id/i + ); + await assert.rejects(live.removeDocument('missing'), /unknown id/i); +});