Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
231 lines (180 loc) · 7.86 KB

ARCHITECTURE.md

File metadata and controls

231 lines (180 loc) · 7.86 KB
Raw

APEX Architecture

Purpose

APEX is the document-intelligence system behind the U.S. Strategy Corpus.

Its role is to convert public federal strategy documents into structured, provenance-backed data that downstream products can rely on. It is not primarily a chat or RAG system. Its primary outputs are durable records, review workflows, and APIs.

System Overview

                public federal strategy documents
                              |
                              v
                    registration + ingestion
                              |
                              v
                     structure extraction layer
                              |
                              v
                    provenance resolution layer
                              |
                              v
                 normalization + enrichment layers
                              |
                              v
                   review UI + corpus API surface
                              |
               +--------------+--------------+
               |                             |
               v                             v
   Performance.gov and public reporting   internal analytics tools

Current Runtime Architecture

+------------------+       +------------------+       +------------------+
| React Frontend   | ----> | FastAPI Backend  | ----> | USAI-compatible  |
| Vite + TS        |       | SQLAlchemy       |       | LLM endpoint     |
+------------------+       +--------+---------+       +------------------+
                                    |
              +---------------------+----------------------+
              |                                            |
              v                                            v
   +----------+-----------+                    +-----------+----------+
   | SQLite / corpus DB   |                    | PDF + object storage |
   | documents, goals,    |                    | local + S3 fallback  |
   | objectives, QA, etc. |                    +----------------------+
   +----------------------+

In local development, APEX uses the repo-local corpus and PDFs. In cloud.gov, the current deployment uses the bundled SQLite corpus plus S3-backed PDF serving.

Product-Critical Design Principles

1. Source Traceability

Every meaningful derived field should point back to source evidence.

Examples:

  • extracted goal -> source page + source text
  • extracted objective -> source page + source text
  • future summary -> citations
  • future relationship inference -> evidence text + confidence

2. Layered Processing

APEX should behave like a document-state pipeline, not a one-shot prompt.

Core stages:

  1. registration
  2. structure extraction
  3. provenance resolution
  4. metadata normalization
  5. document context extraction
  6. summaries with citations
  7. relationship extraction
  8. review and publication

Each stage consumes persisted output from the previous stage.

This is now operational, not aspirational. The production extractor behaves as a multi-stage control loop:

  1. register the document
  2. extract candidate hierarchy
  3. resolve provenance deterministically
  4. run OCR fallback when native parsing appears structurally weak
  5. run AI QA over the extracted result
  6. retry a bounded number of times when the QA findings are repairable
  7. publish as review-ready or hold for human review

3. Fail Closed On Interpretation

Interpretive layers such as summaries or relationship inference should not silently degrade into synthetic fallback text. If the model is unavailable or the result is not trustworthy, the stage should fail and remain incomplete.

4. Corpus First

The core asset is the dataset, not the interface. The UI exists to review, validate, and improve the data layer.

5. Benchmark-Guided Learning

Benchmark comparison is not only an evaluation layer. It is how APEX learned which extraction problems mattered most.

Comparing AI-driven extraction against many historical benchmarked plans surfaced recurring defect classes:

  • TOC contamination
  • heading truncation
  • local numbering resets
  • page anchoring mistakes
  • graphical layout misses
  • post-processing regressions on previously good output

Those benchmark findings led directly to architectural behavior:

  • provenance refresh is conservative and guarded against downgrades
  • OCR fallback is selective
  • QA can block publication
  • reruns are snapshot-backed
  • retries are bounded and stage-aware

Core Data Domains

Documents

Documents represent public strategy artifacts and carry operational metadata:

  • agency
  • document type
  • title
  • file path
  • fiscal period start/end
  • publication year
  • administration
  • ingestion status and source metadata

Structure

The extracted structural layer currently centers on:

  • goals
  • objectives
  • measures for performance-report contexts

Provenance

Goals and objectives now carry provenance-oriented fields such as:

  • source_page
  • source_text
  • source_highlight_text
  • source_heading
  • source_match_method
  • source_confidence

Document context also carries page-aware provenance where available:

  • mission_page
  • vision_page
  • values_pages

Evaluation

When benchmark data exists, APEX compares extracted structure against human-prepared data and reports:

  • precision
  • recall
  • F1
  • matched and unmatched items
  • benchmark applicability/exclusion

APEX also now has a distinct AI-driven extraction QA layer for both benchmarked and unbenchmarked plans. That layer checks:

  • structural consistency
  • truncation
  • numbering
  • provenance/source grounding
  • document context extraction quality

Emerging Enrichment Layers

The next data layers are:

  • document-level mission and vision
  • values when analytically useful
  • grounded summaries with citations
  • agency, organization, and stakeholder mentions
  • stakeholder roles and relationship types
  • inter-goal and interagency relationships
  • temporal continuity across administrations

Review Architecture

The review surface is intentionally plan-centric.

Current design:

  • explore page lists plans, not agencies
  • plan review page shows extracted hierarchy on the left
  • PDF remains fixed on the right for source verification
  • mission, vision, and values appear as first-class extracted items when present
  • benchmark data appears when available
  • agency collaborator tags open evidence-backed detail dialogs

This aligns the product with the review task: check what was extracted against what the document actually says.

Corpus Acquisition Strategy

APEX should support a repeatable acquisition process for the public U.S. strategy corpus.

Recommended acquisition model:

  • maintain a registry of known agency strategy pages and document families
  • fetch or scrape those pages on a schedule
  • detect new PDFs by URL, title, and file hash
  • download into incoming folders
  • register, extract, and review

Important constraints:

  • prefer deterministic and auditable collection over broad crawling
  • store source URL and retrieval metadata
  • preserve exact source files
  • support reprocessing when extraction logic improves

Near-Term Technical Priorities

  1. formalize pipeline state beyond in-memory extraction status
  2. strengthen summary and citation layers, but fail closed when unavailable
  3. expand relationship extraction beyond agency mentions
  4. improve acquisition tooling for new public strategy documents
  5. harden cloud storage patterns beyond PDFs
  6. add extraction-run history as a first-class persistent model

Longer-Term Architecture Direction

The current local stack is appropriate for iteration, but the architecture should evolve toward:

  • durable pipeline status tracking
  • versioned extraction runs
  • better migration tooling
  • stronger acquisition registry and scheduler
  • richer API surface for downstream analytics products

The architecture should continue to optimize for one outcome: a trustworthy system of record for federal strategy.