Incremental updates: 8 issues to fix for production readiness

[jonathanpopham]

Problem

• Incremental sends a changedFiles field the API doesn't understand → 99% of graph nodes get new UUIDs on every single-file update
• API reclassifies domains from scratch on every request → send 1 file, get garbage domains instead of the real domain model
• Merge unconditionally replaces domains with the incremental response → correct domains destroyed after every edit

Solution

• Send small zip through normal endpoint instead of broken AnalyzeIncremental
• Preserve domains from last full generate on incremental — never overwrite them with partial results
• Assign new files to existing domains client-side using directory prefix matching
• Full refresh (analyze/commit) is the only time domains regenerate
• Send existing graph alongside zip so API can preserve domains and only re-parse changed files (supermodeltools/supermodel-public-api#738)
• Domain name stability across full refreshes via LLM prompt seeding (supermodeltools/supermodel-public-api#738)

Action items

Incremental fix (P0)

• Swap `AnalyzeIncremental` → `AnalyzeSidecars` in daemon
• Stop replacing domains on incremental merge
• Delete dead `postIncrementalZip` code

Domain assignment

• Detect new files after merge that have no domain, assign by directory prefix

Tests

• Port graph-fusion's 25+ daemon/merge/render tests into CLI

Polish

• Stop making two API calls in `analyze`
• Setup wizard should offer to start `watch`
• Hook should use absolute path to binary
• Watch should clean sidecars on Ctrl+C
• Fix global map mutation in zip.go
• Hook should warn if daemon isn't running

API integration (depends on supermodeltools/supermodel-public-api#738)

• Send existing graph JSON alongside zip on incremental requests
• Seed LLM prompt with previous domain names on full refresh

[coderabbitai]

🔗 Related PRs

#16 - fix: handle async 202 job polling for analyze [merged]
#18 - feat: add supermodel restore command [merged]
#41 - feat: expand skipDirs to 24 entries, sync across all slices [merged]
#45 - feat: file mode — write graph files from analyze, add watch/clean/hook [closed]
#55 - Show graph stats summary after watch generates initial files [closed]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[jonathanpopham]

P0: Incremental update is fundamentally broken — 99% graph churn on single-file update

Repro

cd ~/cli
go run . clean; rm -rf .supermodel

# Start watch daemon, let it do full generate
go run . watch &
# Wait for "Ready" in logs (~60s)

# Snapshot the graph
cp .supermodel/graph.json /tmp/graph-before.json

# Trigger incremental update for ONE file
echo '{"tool_name":"Edit","tool_input":{"file_path":"internal/cache/cache.go"}}' | go run . hook

# Wait 30s for debounce + API + merge + render

# Diff before vs after
python3 -c "
import json
before = json.load(open('/tmp/graph-before.json'))
after = json.load(open('.supermodel/graph.json'))
bn = {n['id']: n for n in before['graph']['nodes']}
an = {n['id']: n for n in after['graph']['nodes']}
kept = set(an.keys()) & set(bn.keys())
removed = set(bn.keys()) - set(an.keys())
added = set(an.keys()) - set(bn.keys())
print(f'Before: {len(bn)} nodes')
print(f'After:  {len(an)} nodes')
print(f'Kept (same ID): {len(kept)}')
print(f'Removed: {len(removed)}')
print(f'Added: {len(added)}')
print(f'Churn: {100*len(removed)//len(bn)}%')
"

Results

Metric	Expected	Actual
Nodes churned	~8 (the cache.go nodes)	1797 removed + 1814 added (99%)
IDs preserved	~1793	4 (2 Domain + 2 Subdomain nodes)
Same entity, new UUID	~8	1795

Only 4 nodes survived with their original IDs: Domain:SupermodelAPI, Domain:ArchDocs, Subdomain:Client, Subdomain:PSSG.

Sidecar corruption after merge

Before incremental update, cache.go sidecar correctly shows:

// imports     internal/api/client.go
// imports     internal/api/doc.go
// imports     internal/api/types.go
// imports     internal/config/config.go
// imports     internal/config/doc.go

After incremental update, the sidecar says:

// imports     internal/archdocs/pssg/config/config.go

This is wrong. cache.go imports internal/api and internal/config, not internal/archdocs/pssg/config. The suffix matcher resolved config to the wrong config.go file when the entire graph was churned with new UUIDs.

Domain names also drifted: before had AnalysisSlices · ArchDocs · SharedKernel · SidecarSystem · SupermodelAPI, after had ArchDocs · Command · FileSystemProvider · Kernel · SupermodelAPI.

Root cause (needs investigation)

The API returned the entire graph with new UUIDs for all nodes — not a subgraph for just the changed file. This means either:

1. API doesn't support true incremental — /v1/graphs/supermodel always returns the full graph regardless of changedFiles
2. Client is calling the wrong endpoint — AnalyzeIncremental might not be sending changedFiles correctly
3. changedFiles field is ignored server-side

The merge logic in daemon.mergeGraph() then treats every node as new (because every UUID changed), removes almost all old nodes, adds all new ones, and the suffix-based dependency resolution (@/components/ui/button → matching file path) produces wrong matches when the entire graph is being re-resolved simultaneously.

Impact

• Every incremental update silently corrupts the graph and produces wrong sidecars
• Worse than no incremental updates — the user thinks it's working while dependencies are wrong
• The "2.5x faster, correct fix vs broken fix" A/B results Abraham measured may not hold if the underlying graph is being trashed on every edit

Suggested fix

1. Verify what the API actually returns for AnalyzeIncremental — is it a subgraph or the full graph?
2. If full graph: either fix the API to return a subgraph, or change the merge to do a full replacement (skip the merge entirely, just use the new graph)
3. If subgraph: the changedFiles multipart field may not be reaching the server — check postIncrementalZip serialization
4. Regardless: add a test that verifies node ID stability after incremental update (the graph-fusion TestMergeGraph_UUIDRemapping test covers this, but only with synthetic data — need an integration-level check)

[jonathanpopham]

Context from the archived supermodeltools/supermodel repo — how incremental updates actually worked

The original system (VS Code extension + Java API + Node worker) had a working incremental update pipeline. Here's how it worked, and what it tells us about the current bug.

How the old system worked

The secret was delta.json embedded in the zip. No separate changedFiles multipart field. The manifest went inside the zip:

{
  "upserts": [{"path": "src/Main.ts", "contentHash": "abc123", "baseHash": "def456"}],
  "deletes": ["src/Old.ts"],
  "renames": [{"from": "src/OldName.ts", "to": "src/NewName.ts"}],
  "runId": "repo-uuid-timestamp",
  "parserVersion": "tree-sitter",
  "pipelineVersion": "1"
}

The Java backend (RepoService.java) detected incremental mode by checking if delta.json existed in the zip:

• delta.json present → incremental: apply only the changes to the stored repo state via DeltaZipApplier, re-parse only changed files
• delta.json absent → full: replace everything, parse from scratch

The VS Code extension (fileDeltaWatcher.ts) tracked three things: upserts (new/modified files with content hashes), deletes, and renames. It debounced changes (2-8s window), built a partial zip with just the changed files + the manifest, and uploaded it to POST /repos/{repoId}/upload.

After upload, the Node worker parsed only the changed files (filePaths parameter in the parse request), updated the graph incrementally, and the extension compared the graph before/after to highlight newly created nodes.

Key design details

• No separate query parameter or multipart field for changed files — the delta manifest was embedded in the zip itself
• Backward compatible — missing delta.json meant full replacement, not incremental
• Path validation — DeltaZipApplier only extracted files explicitly listed in the manifest's upsert list (whitelist approach, prevents traversal attacks)
• Hash tracking — each upsert carried contentHash (current) and baseHash (previous) for verification
• Stateful — DeltaManifestRegistry (a ConcurrentHashMap<String, DeltaManifest>) stored pending manifests keyed by repo UUID for downstream progressive classification

Source files: DeltaManifest.java, DeltaManifestRegistry.java, DeltaZipApplier.java, fileDeltaWatcher.ts, uploadService.ts — all in the archived supermodeltools/supermodel repo on the staging branch.

What this tells us about the current bug

The current /v1/graphs/supermodel endpoint doesn't know about changedFiles as a multipart field — that concept never existed in the original system. The API decides incremental vs full based on what's in the zip, not metadata alongside it.

Graph-fusion got this right by accident — it just sends a tiny zip with 1 file, no metadata. The API sees 1 file, processes 1 file, returns a subgraph. We verified this: graph-fusion incremental update for cache.go churned exactly 8 nodes (0%), while the CLI churned 1797 (99%).

The CLI's postIncrementalZip adding a changedFiles multipart field is either ignored by the API (and something else causes the full-graph return) or it confuses the server into treating the request differently. Either way, graph-fusion proves the fix: just send the small zip, nothing extra.

Suggested fix

Replace AnalyzeIncremental (which uses postIncrementalZip with the extra changedFiles field) with AnalyzeSidecars (which uses plain postZip). The daemon already creates a zip with only the changed files — the zip content is the delta. No metadata field needed.

[jonathanpopham]

Why the original delta system matters — domain classification is stateful

Further investigation reveals that even graph-fusion's "working" incremental (0% node churn) silently corrupts the domain classification.

Evidence

Sent 1 file (cache.go) via graph-fusion incremental. Node churn was correct (0%), but:

	Before	After
Domains	CommandCLI, ApiClient, SidecarEngine, ArchDocsGenerator, SharedKernel (5 domains, 20 keyFiles, 15 subdomains)	LocalCache, GraphAPI, CLIConfig (3 domains, 3 keyFiles, 1 subdomain)
belongsTo rels	1064	1063

The API only saw 1 file. It classified domains based on that 1 file and returned 3 tiny domains. The merge then replaced all domains with the new ones (d.ir.Domains = incremental.Domains in daemon.go:479).

This means every sidecar's [impact] section gets wrong domain tags after any incremental update. The "domains" and "risk" scores in sidecars are derived from FileDomain which is populated from the domains array.

Why the original delta system was built

The archived supermodeltools/supermodel repo had delta.json for a reason — the domain classification system needs to know:

1. What changed (upserts) — re-classify only these files
2. What was deleted (deletes) — remove from the graph, don't re-classify
3. What was renamed (renames) — update paths, don't re-classify
4. What didn't change — preserve existing classification

Without this, the API re-runs classification from scratch on whatever files it sees. With 1 file, it produces a 1-file domain model. With 136 files, it produces the correct 5-domain model. There's no in-between.

The specificFilePaths support already exists internally in codegraph.service.js — it just needs to be wired to the HTTP layer so the API knows "process these files for code graph, but DON'T re-run domain classification."

Impact on the "just send a small zip" fix

Option A (drop changedFiles, send small zip) fixes the node churn (P0) but introduces domain drift. Three approaches:

1. Quick fix: On incremental, don't merge domains — keep the original domains from the full generate. This means d.ir.Domains = incremental.Domains should be guarded: only replace if the incremental response has >= the original domain count (or skip domain replacement entirely for incremental).

2. Medium fix: Split the merge — use incremental nodes/relationships (they're correct), but always preserve domains from the last full generate.

3. Proper fix: Add delta.json support to the API data plane, wire specificFilePaths through so incremental updates skip domain reclassification.

The quick fix is probably the right PR for now — it preserves domain correctness while fixing the 99% node churn.

[jonathanpopham]

Design direction: full repo context on every request, no server state

Why the legacy system got it right

The archived supermodeltools/supermodel system worked flawlessly because the server stored the full repo on disk. When a delta zip arrived, DeltaZipApplier patched the stored copy, then the pipeline re-ran against the full repo with patches applied. The LLM domain classifier always saw the complete codebase. It just didn't have to re-parse every file — specificFilePaths told the codegraph builder which files to re-parse.

The current system gives the classifier whatever's in the zip. Send 1 file, it invents 3 tiny domains. Send 136 files, it produces the correct 5-domain model. Domain classification requires full repo context — there's no way around this.

Why "smart client, dumb server" breaks down

We explored Option 3 (preserve domains client-side, assign new files to existing domains using deterministic logic, periodic full refresh on commit). The problem: sometimes you need to add domains. New nodes need domains and get sorted into them. The deterministic assignment logic (directory prefix + import affinity) can assign to existing domains, but when a new module doesn't fit any existing domain, you need the LLM — and the LLM needs to see the full repo.

The fix: flip the legacy architecture

The legacy system had the server hold state. But the CLI is the state — the repo is right there on disk. Flip it:

Send the full zip every time, but tell the server which files to re-parse.

Edit cache.go
  → git archive HEAD (~100ms, full repo)
  → Embed delta.json in zip: {"upserts": ["internal/cache/cache.go"]}
  → POST /v1/graphs/supermodel (full zip, ~225KB)
  → Server extracts full zip (has full repo context)
  → Server reads delta.json → passes specificFilePaths to codegraph (re-parse cache.go only)
  → Domain classifier sees full repo → correct domains
  → Returns updated graph with correct domains + updated code graph for changed files

Performance

The expensive part of the pipeline is the LLM call for domain classification (~10-30s). But most edits don't change domain structure — editing a function body, fixing a bug, adding a method. The server can skip the LLM call when the delta only contains upserts to existing files (no new directories, no structural changes) and reuse the deterministic assignment logic against existing domains.

When the LLM needs to re-run:

• New top-level directory or module
• File structure fundamentally reorganized
• User explicitly forces full refresh

Step	Time
git archive HEAD	~100ms
Upload ~225KB zip	~100ms
Server: extract + codegraph for 1 file (skip LLM)	~5s
Client: merge + render sidecars	~200ms
Total	~5-6s per edit

vs ~30-60s for the current full pipeline with LLM domain classification.

What exists already

The pieces are in the codebase — they just need wiring:

• specificFilePaths support in codegraph.service.js — processes only specified files instead of scanning the entire directory. Built but not exposed through the HTTP endpoint.
• AiFileDomainSuggesterService — takes a single file + its imports, checks against existing domains, returns useExistingDomain (boolean) or newDomainName (string). Can suggest new domain creation. Built but not wired into any endpoint.
• ClassificationOrchestratorService — deterministic file-to-domain assignment via directory prefix matching and import affinity. Already runs after LLM domain generation. Could be re-run independently without the LLM for incremental edits.
• delta.json detection pattern — the legacy DeltaZipApplier checked for delta.json in the zip root. Same pattern can be added to the current ZipHydratorService.
• git archive HEAD — already used in the CLI's internal/archive/ package. Fast (~100ms), respects .gitignore, produces compressed output.

What needs to change

API (data plane):

1. ZipHydratorService: detect delta.json in the zip, extract it as metadata
2. Controller: when delta present, pass specificFilePaths to codegraph.service.js
3. Controller: when delta is edit-only (upserts to existing files, no structural changes), skip LLM domain classification — run ClassificationOrchestratorService against previous domain structure instead
4. Wire AiFileDomainSuggesterService for the "new file doesn't fit any domain" edge case

CLI:

1. Daemon: on incremental, create full git archive zip instead of partial zip
2. Daemon: embed delta.json manifest in the zip with upserts/deletes/renames
3. Use AnalyzeSidecars (normal endpoint) instead of AnalyzeIncremental — the delta.json in the zip tells the server what changed
4. Merge logic: can be simplified since server returns correct domains — may not need the complex UUID remapping if server returns a full updated graph

Summary

The legacy system was right: domain classification needs full repo context. The mistake was putting the state on the server. The repo lives on the client — send it every time, tell the server what changed, let the server skip the expensive parts. No server state, correct domains, ~5s incremental updates.

[jonathanpopham]

TL;DR for the team

Current state

Incremental updates are broken in the CLI. When the daemon processes a single file change, 99% of the graph gets new UUIDs and the domain classification is replaced with garbage (5 correct domains → 3 wrong ones). Sidecars render with wrong dependency and domain data after any incremental update. Graph-fusion doesn't have this specific bug, but it has its own version of the domain problem — domains drift because the API only sees the 1 changed file.

Root cause: the API classifies domains based on what's in the zip. Send it 1 file, it invents domains for 1 file. Domain classification needs full repo context to be correct.

The plan

Keep the API stateless. Send the full repo on every incremental update, but embed a delta.json manifest in the zip that tells the server which files actually changed.

What the server does differently when it sees delta.json:

• Code graph: only re-parse the changed files (specificFilePaths — already built in codegraph.service.js, just needs wiring)
• Domains: skip the LLM call for simple edits, re-run the deterministic assignment logic against existing domains
• Result: correct domains + updated code graph, in ~5s instead of ~30-60s

What the CLI does:

• git archive HEAD on every edit (~100ms) instead of zipping 1 file
• Embed delta.json: {"upserts": ["src/foo.ts"]} in the zip
• Normal merge + render on the response

Why this works: the repo is right there on disk — there's no reason to maintain server-side state when the client can send the full context every time. The delta manifest just tells the server what to skip, not what to store.

This is the same pattern as the original supermodeltools/supermodel system, but with the client as the source of truth instead of the server. No server state, no cache invalidation, no concurrency issues.

Work needed

• API: wire specificFilePaths to the HTTP layer, detect delta.json in zip, skip LLM when delta is edit-only
• CLI: send full zip + delta.json instead of partial zip, drop the broken changedFiles multipart field

Full investigation and data are in the comments above.

[jonathanpopham]

Why preserving domains on incremental is correct

A domain is a semantic grouping. "Authentication", "Billing", "APILayer" — it's what area of the system a piece of code belongs to.

A file gets assigned to a domain based on where it is (directory prefix) and what it imports (import affinity). Not based on what the functions inside it do.

So when does a file's domain actually change?

• I edit internal/cache/cache.go — add a function, fix a bug, refactor. Domain doesn't change. Same directory, same imports, same purpose.
• I create internal/cache/lru.go — new file in an existing directory. Same domain as its neighbors. Directory prefix matching handles this.
• I create internal/notifications/slack.go — new module entirely. Might need a new domain. But this is a structural change, not a normal edit.
• I move a file from one package to another — domain might change. Also a structural change.

The vast majority of incremental updates are case 1 and 2. The file doesn't move. Its imports don't fundamentally change. Its domain doesn't change.

Domain assignment is a function of repo structure, not file content. And repo structure doesn't change on most edits.

So preserving domains from the last full generate is actually correct for incremental updates. The domain only needs to update when the repo structure changes — new directories, deleted modules, moved files. Those are exactly the cases where a full refresh makes sense anyway.

The only gap is case 2 — new file in an existing directory. It needs to be assigned to the same domain as its neighbors. That's the deterministic directory prefix logic. The client has everything it needs to do that locally.

[jonathanpopham]

Design Plan: Incremental Updates

Current State

The incremental pipeline exists but is broken in two ways:

1. 99% graph churn (P0). The CLI's AnalyzeIncremental sends a changedFiles multipart field that the API doesn't understand. The API returns a full graph with new UUIDs for every node. The merge logic then churns the entire graph. Graph-fusion doesn't have this bug — it sends a small zip through the normal endpoint and gets back a correct subgraph.

2. Domain corruption. Even with correct node merging (graph-fusion), the API reclassifies domains from scratch based on whatever's in the zip. Send 1 file, get domains for 1 file. The merge unconditionally replaces all domains with the incremental response (d.ir.Domains = incremental.Domains), destroying the correct domain model.

Desired State

1. supermodel analyze or supermodel watch does a full generate — correct graph, correct domains
2. Agent edits a file → hook fires → daemon sends small zip → API returns subgraph → client merges nodes/rels, preserves domains → sidecars re-render in ~5s
3. New file created in existing directory → client assigns it to the same domain as its neighbors (deterministic, no API call)
4. Structural changes (new module, major refactor) → next full refresh produces fresh domains
5. Domain names stable across full refreshes (future, requires API prompt seeding)

The Delta

Phase 1: Fix the P0 (CLI only, ~15 lines)

• internal/files/daemon.go: replace d.client.AnalyzeIncremental() with d.client.AnalyzeSidecars() in incrementalUpdate. Drop the changedFiles parameter.
• internal/files/daemon.go: in mergeGraph, never replace domains on incremental. Preserve domains from the last full generate.
• internal/api/client.go: AnalyzeIncremental and postIncrementalZip become dead code (can delete).

Fixes both the 99% churn and domain corruption. Graph-fusion proves this approach works (0% churn, correct subgraph).

Phase 2: Client-side domain assignment for new files (CLI only, ~50-80 lines)

• internal/files/daemon.go: after merging incremental nodes, detect new files without domain assignments.
• internal/files/graph.go: add AssignNewFileToDomain() — port the deterministic logic from ClassificationOrchestratorService: longest directory prefix match against domain keyFiles, import affinity as tiebreaker.

Handles "new file in existing directory" without an API call.

Phase 3: Port tests from graph-fusion (CLI only, ~800-1000 lines)

• internal/files/daemon_test.go: port 25+ tests from graph-fusion — merge (basic, file replacement, UUID remapping, external dep resolution, empty incremental, concurrency), writeStatus, saveCache, loadOrGenerate, UDP listener, watcher forwarding.
• internal/files/graph_test.go: port from graph-fusion.
• internal/files/render_test.go: port from graph-fusion.

Phase 4: Fix remaining issues (CLI only, ~95 lines total)

• Double API call: cmd/analyze.go — pass graph from analyze.Run() to files.Generate() instead of calling API twice. (~20 lines)
• Setup runs analyze not watch: internal/setup/wizard.go — offer "Start watcher?" after analyze. (~15 lines)
• Hook PATH: internal/setup/wizard.go — resolve absolute path to binary, warn if not on PATH. (~20 lines)
• Watch cleanup on Ctrl+C: cmd/watch.go — signal handler, call files.Clean() after cancellation. (~15 lines)
• Global map mutation: internal/files/zip.go — clone maps before mutating in loadCustomExclusions. (~10 lines)
• Daemon health check: internal/files/handler.go — check .supermodel/status age in Hook(), warn if stale. (~15 lines)

Phase 5: Domain name stability (API + CLI, separate issue)

• API domain-analysis.service.smart-sampling.js: accept optional previousDomains, seed the LLM prompt to reuse names where groupings still apply.
• CLI: on full generate, send cached domain names alongside the zip.
• Requires prompt engineering iteration. File as separate issue.

Tradeoffs

What this gets right:

• API stays completely stateless
• Small zip on every edit (~1KB), not the full repo
• Nodes and relationships update correctly in ~5s
• Domains preserved (correct) during incremental edits
• New files get domain assignment client-side
• No new API endpoints needed for Phases 1-4

What this accepts as limitations:

• Domains are eventually consistent — fresh domains require a full refresh
• New files that don't fit ANY existing domain get assigned by heuristic (directory prefix). Wrong until next full refresh.
• Domain names may drift across full refreshes (Phase 5 addresses this)
• Deletes/renames leave phantom nodes until next full refresh

What we chose not to do:

• No server-side state (DeltaManifestRegistry, stored repos)
• No full repo upload on every edit (too expensive for 20MB+ repos)
• No delta.json protocol (would require significant data plane work)
• No client-side LLM calls

Sequencing

Phase 1 is the only blocker — fixes the P0, ships as a single PR (~15 lines). Phases 2-4 are independent and can be parallelized. Phase 5 is a separate project on the data plane.

[jonathanpopham]

Action items

• Swap AnalyzeIncremental → AnalyzeSidecars in daemon
• Stop replacing domains on incremental merge
• Delete dead postIncrementalZip code
• Detect new files after merge that have no domain, assign by directory prefix
• Port graph-fusion's 25+ daemon/merge/render tests into CLI
• Stop making two API calls in analyze
• Setup wizard should offer to start watch
• Hook should use absolute path to binary
• Watch should clean sidecars on Ctrl+C
• Fix global map mutation in zip.go
• Hook should warn if daemon isn't running
• Seed LLM prompt with previous domain names on full refresh (API work, separate issue)

[jonathanpopham]

Reopening — PR #62 fixed the P0 (incremental churn + domain corruption) but this issue tracks additional work:

Still open:

• PR fix: eliminate double API call in analyze when file mode is on #63: eliminate double API call in analyze (open, ready for review)
• Port remaining tests from graph-fusion (render, graph indexing)
• API: accept existing graph for domain preservation (supermodeltools/supermodel-public-api#738)

Done:

• Swap AnalyzeIncremental → AnalyzeSidecars in daemon (PR fix: use normal endpoint for incremental updates, preserve domains #62, merged)
• Stop replacing domains on incremental merge (PR fix: use normal endpoint for incremental updates, preserve domains #62, merged)
• Delete dead postIncrementalZip code (PR fix: use normal endpoint for incremental updates, preserve domains #62, merged)
• 9 merge + domain preservation tests added (PR fix: use normal endpoint for incremental updates, preserve domains #62, merged)