Plan 002: Cache Parquet data, indexes, and domain config in the frontend instead of reloading per request

[strickvl]

Plan 002: Cache Parquet data, indexes, and domain config in the frontend instead of reloading per request

Executor instructions: Follow this plan step by step. Run every
verification command and confirm the expected result before moving to the
next step. If anything in the "STOP conditions" section occurs, stop and
report — do not improvise. When done, update the status row for this plan
in plans/README.md.

Drift check (run first): git diff --stat 5b5c634..HEAD -- src/frontend/ src/config_loader.py
If any in-scope file changed since this plan was written, compare the
"Current state" excerpts against the live code before proceeding; on a
mismatch, treat it as a STOP condition.

Status

• Priority: P1
• Effort: S
• Risk: LOW
• Depends on: none
• Category: perf
• Planned at: commit 5b5c634, 2026-06-12

Why this matters

Every page view in the FastHTML web UI re-reads all four entity Parquet files
from disk, converts them to Python dicts, rebuilds all lookup indexes, and
re-parses the domain's YAML config. A list page followed by a detail click
does all of that twice. The data only changes when the extraction pipeline
writes new tables, so almost all of this work is wasted. After this plan:
domain data and indexes are cached in memory keyed by the Parquet files'
modification times (so a fresh pipeline run is picked up automatically), and
DomainConfig instances are created once per domain instead of per request.

Current state

• src/frontend/data_access.py — data loading layer.

  • load_parquet(path) (line 18) reads a Parquet file with PyArrow,
    returns [] if the file doesn't exist.
  • get_domain_data(domain) (line 39) creates a new DomainConfig,
    resolves config.get_output_dir(), and loads people.parquet,
    events.parquet, locations.parquet, organizations.parquet fresh on
    every call. On any exception it falls back to legacy data/entities
    paths.
  • build_indexes(domain_data) (line 179) builds four {key: entity}
    dicts via make_person_key / make_event_key / make_location_key /
    make_org_key.
  • Module-level defaults exist for backward compatibility but routes do not
    use them (lines 88–92, 204–208: _default_data = get_domain_data(),
    _default_indexes = build_indexes(_default_data)).

• Route handlers call these per request. Example
  (src/frontend/routes/people.py:124-130):

  @rt("/people")
  def list_people(request):
      error_handler = ErrorHandler("people_list", {"route": "/people"})
      try:
          current_domain = get_current_domain(request)
          domain_data = get_domain_data(current_domain)

  Find every call site with: grep -rn "get_domain_data(\|build_indexes(" src/frontend/
  (expect ~2 get_domain_data + ~2 build_indexes calls per entity route
  file: people, organizations, locations, events — list + detail views).

• src/config_loader.py — DomainConfig.__init__ (line 19) sets
  self.config_dir = f"configs/{domain}" unless the class attribute
  DomainConfig.config_dir overrides it (a test hook — preserve this).
  load_config() (line 56) has @lru_cache(maxsize=None) on the instance
  method, so a new instance per request means a cold cache per request.
  There is a B019 per-file ruff ignore for this in pyproject.toml
  ("lru_cache on methods is intentional; instances are module-cached") — the
  module-caching it mentions is what this plan actually introduces.

• Several route files also construct DomainConfig(current_domain) directly;
  find them with: grep -rn "DomainConfig(" src/frontend/.

• Conventions: typing.Dict / typing.Tuple over builtins; frontend test
  pattern in tests/test_frontend_versioning.py.

Commands you will need

Purpose	Command	Expected on success
Tests	just test	all pass
New tests only	uv run pytest tests/test_frontend_caching.py -v	all pass
Lint+format	just format && just lint	exit 0
Full CI parity	just ci	exit 0
Manual smoke	just frontend then open http://localhost:5001	pages render

Scope

In scope (the only files you should modify/create):

• src/frontend/data_access.py
• src/config_loader.py (add a factory function only — do not change DomainConfig behavior)
• src/frontend/routes/people.py, organizations.py, locations.py, events.py, home.py (call-site swaps only)
• src/frontend/app_config.py (only if it constructs DomainConfig directly)
• tests/test_frontend_caching.py (create)

Out of scope (do NOT touch):

• src/process_and_extract.py and src/engine/** — the pipeline constructs
  its own DomainConfig and passes it down; it does not need this cache.
• Filtering/rendering logic inside the routes — only swap the data-access calls.
• The module-level _default_data / people_data etc. exports in
  data_access.py — leave them (other modules may import them).

Git workflow

• Branch: advisor/002-frontend-caching
• Commit style: imperative subject, backticks around code identifiers.
• Do NOT push or open a PR unless the operator instructed it.

Steps

Step 1: Add a cached domain-config factory

In src/config_loader.py, add at module level (after the DomainConfig
class):

@lru_cache(maxsize=None)
def get_domain_config(domain: str) -> "DomainConfig":
    """Return a shared DomainConfig per domain (config is immutable at runtime)."""
    return DomainConfig(domain)

(lru_cache is already imported in this module.) Do not change
DomainConfig itself — the DomainConfig.config_dir class-attribute test
hook must keep working; note that tests using that hook should call
get_domain_config.cache_clear(), which your new tests will do.

Verify: uv run python -c "from src.config_loader import get_domain_config; a=get_domain_config('guantanamo'); b=get_domain_config('guantanamo'); print(a is b)" → True

Step 2: Cache domain data + indexes by Parquet mtime signature

In src/frontend/data_access.py:

1. Switch the DomainConfig(domain) call inside get_domain_data to
   get_domain_config(domain) (import it from src.config_loader).
2. Add an mtime-keyed cache and an index accessor:

_domain_cache: Dict[str, Tuple[Tuple[float, ...], Dict[str, Any]]] = {}

def _mtime_signature(paths: List[str]) -> Tuple[float, ...]:
    return tuple(os.path.getmtime(p) if os.path.exists(p) else 0.0 for p in paths)

def get_domain_bundle(domain: str = "guantanamo") -> Dict[str, Any]:
    """Return {'data': ..., 'indexes': ...}, cached until any Parquet file changes."""
    # resolve the four file paths exactly as get_domain_data does today
    sig = _mtime_signature([people_file, events_file, locations_file, orgs_file])
    cached = _domain_cache.get(domain)
    if cached and cached[0] == sig:
        return cached[1]
    data = ...      # the existing loading logic
    bundle = {"data": data, "indexes": build_indexes(data)}
    _domain_cache[domain] = (sig, bundle)
    return bundle

Restructure so get_domain_data(domain) returns
get_domain_bundle(domain)["data"] and a new
get_domain_indexes(domain) returns ...["indexes"] — existing callers
of get_domain_data keep working unchanged. Keep the legacy
data/entities fallback inside the loading logic, with its own signature
computed from the fallback paths.

Verify: uv run pytest tests/ -k frontend -v → existing frontend tests still pass

Step 3: Swap route call sites

• Replace each build_indexes(domain_data) call in
  src/frontend/routes/*.py with get_domain_indexes(current_domain)
  (adjust imports; the authoritative list comes from
  grep -rn "build_indexes(" src/frontend/routes/).
• Replace each direct DomainConfig(...) construction in src/frontend/
  with get_domain_config(...) (list from grep -rn "DomainConfig(" src/frontend/).

Verify: grep -rn "build_indexes(" src/frontend/routes/ → no matches;
grep -rn "DomainConfig(" src/frontend/routes/ src/frontend/app_config.py → no matches

Step 4: Tests

Create tests/test_frontend_caching.py. Pattern: patch
src.frontend.data_access.load_parquet with a counting fake (it is defined
in that module), and patch src.frontend.data_access.get_domain_config (as
imported there) with a stub whose get_output_dir() returns a tmp_path
directory. Then:

1. Cache hit: call get_domain_data("x") twice with unchanged files →
   counting fake invoked exactly 4 times total (once per entity file, first
   call only).
2. Invalidation on mtime change: create one real (tiny) Parquet file in
   tmp_path with pyarrow (pa.Table.from_pylist([{"name": "A"}])), call
   once, then os.utime(path, (t+10, t+10)), call again → loader invoked
   again.
3. Indexes cached with data: get_domain_indexes("x") twice → index
   dicts are the same object (is).

Call get_domain_config.cache_clear() and clear _domain_cache in a fixture
so tests are order-independent.

Verify: uv run pytest tests/test_frontend_caching.py -v → all pass

Step 5: Full suite + manual smoke

Run the suite, then just frontend and click through: home → People list →
a person detail → Events list. Pages must render with data (use the
guantanamo domain if its Parquet files exist locally; if no data files exist,
empty lists rendering without error is the expected behavior).

Verify: just ci → exit 0

Test plan

Covered in Step 4. Model fixture/mock style on
tests/test_frontend_versioning.py. Three new tests minimum: cache hit,
mtime invalidation, shared index identity.

Done criteria

• just ci exits 0
• uv run pytest tests/test_frontend_caching.py -v → 3+ new tests pass
• grep -rn "build_indexes(" src/frontend/routes/ → no matches
• grep -rn "DomainConfig(" src/frontend/routes/ → no matches
• No files outside the in-scope list modified (git status)
• plans/README.md status row updated

STOP conditions

Stop and report back (do not improvise) if:

• get_domain_data in data_access.py no longer matches the excerpt
  (someone added caching already).
• Any route mutates the structures returned by get_domain_data /
  build_indexes (grep the routes for .append(, .pop(, del on
  domain_data / index variables). Shared cached objects must not be
  mutated per request — report which route does it.
• Removing per-request DomainConfig construction breaks a route because it
  relied on the constructor's _validate_domain() raising for bad domains —
  check how invalid ?domain= values are handled in
  get_current_domain (src/frontend/app_config.py:58) before assuming.

Maintenance notes

• The cache key is file mtimes. If entity tables ever move to partitioned
  directories (multiple files per type), _mtime_signature must walk them.
• If a future plan adds in-app editing of entities (a known direction idea),
  writes must call into data_access to invalidate or update _domain_cache.
• Reviewer should scrutinize: that no route mutates cached lists/dicts, and
  that the legacy fallback path still works when a domain config is missing.