Hypermedia based object models that can be progressively enhanced by custom elements

[shah]

We need a repeatable pattern for building complex UI components where all data and baseline structure are rendered by the server as strict HTML (or XML embedded in HTML), and client-side JavaScript is limited to progressive enhancement via ESM custom elements. The <natural-grid> implementation in lib/tabular-ux/grid.js is the reference model. This ticket generalizes that model into a reusable approach for almost any complex UI, while preserving a server-first, deterministic, AI-maintainable architecture.

Client-heavy component frameworks tend to blur three concerns: data, structure, and behavior. That makes pages harder to validate, harder to secure, harder to audit, and harder to maintain when AI agents are involved. We want the inverse: always start with a complete, meaningful document that works without JavaScript, then progressively enhance only the behaviors that require client runtime.

The core design principle is: the server owns the truth. The browser enhances the experience.

Goals:

1. All pages ship as complete HTML from the server (no “blank app shell” patterns).
2. Components express their data in strict HTML (tables, lists, forms) or in a non-executable data block embedded within HTML (XML or JSON in inert containers).
3. Behaviors are added via modern ESM custom elements, with open ShadowRoot only when needed, and with deterministic styling and plugin pipelines where applicable.
4. The enhancement is optional. Without JS, the HTML should still be readable, navigable, and usable.
5. The pattern is easy for humans and AI maintainers to extend without forking a core.

Non-goals:

• Building a new SPA framework. Do not create new frameworks, rely on HTML (and XML if necessary), CSS and modern, JSDocs-typed JavaScript.
• Moving server rendering logic into client templates. Use SSE for interactivity where necessary but keep things server-first when possible.
• Requiring a build step for every UI feature. Use AI to maintain modern JSDocs-typed JavaScript and CSS without any build tools.
• Using custom elements as a “framework replacement” instead of as progressive enhancement. Don't overcomplicate.

Use the module JSDoc for @module lib/tabular-ux/grid.js as the canonical reference for how a complex UX is delivered as <natural-grid>:

• multi-source configuration resolution
• data provider abstraction
• plugin pipeline with stable ordering
• deterministic styling pipeline
• a single internal mutable model
• virtualization support
• security notes for HTML payloads
• attribute-driven re-initialization behavior

Deliverable 1: General pattern document and scaffolding. Create a short architecture doc (e.g. docs/hypermedia-object-models.md) that generalizes the <natural-grid> approach into a template for new components. The doc must define the standard “Hypermedia Object Model” pattern:

A. Server-rendered baseline markup (the object model)

• The server emits a complete DOM structure that represents the component’s data model in strict HTML, such as:

  • <table> for tabular
  • <dl> for key-value object views
  • <ul>/<ol> for collections
  • <form> for editable models
  • <article>/<section> with semantic headings for documents

• Optionally embed richer structured data using inert containers:

  • <template> containing XML (preferred when structure matters)
  • <script type="application/json"> (acceptable when structure is simple)
  • <data> nodes or data-* attributes for small scalars

• The baseline should be accessible and meaningful with zero JS.

B. Progressive enhancement custom element (behavior layer)

• A custom element upgrades the baseline:

  • attaches events, adds toolbars, virtualization, inline editing, sorting/filtering, pagination, etc.
  • must never require that the server omit markup

• Enhancement modes must be deterministic and controlled by attributes and/or embedded configuration blocks.

C. Configuration resolution rules (generalized from NaturalGridSpec)
Define a consistent order and naming for config sources across all components, similar to <natural-grid>:

1. inline inert config inside the element (preferred)
2. external inert config referenced by id
3. “upgrade” an existing strict HTML structure by id (external upgrade)
4. upgrade inline strict HTML structure (inline upgrade)
5. optional factory hook for advanced cases

Provide generic attribute conventions:

• data-init="xml|json|upgrade|factory"
• data-config-id="..."
• data-source-id="..." (points at the strict HTML model to upgrade)
• data-factory="..."

D. Data model approach

• Keep one in-memory model per component instance
• Prefer a clear “snapshot” shape plus optional incremental updates
• Define a consistent internal status model (init|loading|ready|error)
• Make rendering re-entrant and idempotent (render can be called repeatedly safely)

E. Plugin pipeline as an extension mechanism
Generalize the <natural-grid> plugin approach into a standard component plugin pipeline:

• presentation plugins (styles/theme)
• content-supplier plugins (data adapters/providers)
• functionality plugins (actions/behaviors/renderers)

Define a shared minimal plugin API concept:

• addStyles(cssText, meta) ordered deterministically
• on/emit internal event bus (not DOM events by default)
• registration hooks (data providers, renderers, actions)
• requestRender, requestDataRefresh, mutation helpers

F. Deterministic styling pipeline

• Base styles first
• Optional theme styles
• Plugin styles ordered by stable (priority, id)
• Optional user styles last
• Use adoptedStyleSheets when available, fallback to injected <style>

G. Security rules

• Default to text rendering for values
• If allowing HTML payloads, require explicit opt-in and document that sanitization is upstream responsibility
• Prefer server-side sanitization and/or allowlist-based transforms

Deliverable 2: Build a generalized base module for custom elements. Create a lightweight shared base utility module (names are suggestions):

• lib/hypermedia-ux/ce_base.ts or lib/hypermedia-ux/custom_element_base.ts
  Include utilities that multiple components can reuse:
• config discovery helpers (inline, external by id, upgrade existing markup)
• deterministic style injection manager
• simple event bus utility
• stable plugin loader and normalization (import() + metadata extraction)
• guardrails for re-init on attribute changes
• “upgrade from strict HTML” helpers (like gridSpecFromTable but generalized)

Deliverable 3: Provide a second example component besides <natural-grid>. Implement one additional custom element that proves the pattern generalizes, for example:

• <natural-object>: upgrades a <dl> or a <table> key-value structure into a richer inspector (collapsible sections, copy buttons, diff highlighting)
  or
• <natural-form>: upgrades a server-rendered <form> into enhanced validation, async submit, and field-level hints
  or
• <natural-timeline>: upgrades a server-rendered <ol> into filtering and progressive disclosure

Requirements for the second component:

• server emits strict HTML baseline

• element upgrades inline or external baseline

• uses the shared base module utilities

• demonstrates at least:

  • config resolution order
  • deterministic styling injection
  • a minimal plugin hook (even if only one built-in plugin)
  • re-init on one observed attribute change

Acceptance criteria:

1. Architecture doc exists and clearly specifies the pattern and conventions.
2. <natural-grid> remains the reference but the new base module extracts reusable primitives without breaking the grid.
3. A second component exists and follows the same conventions.
4. All examples work with JS disabled (baseline still renders).
5. Custom elements load via ESM with no bundler requirement.
6. Styling and plugin ordering are deterministic.
7. Security behavior is documented, with safe defaults.

Implementation notes and guardrails:

• Prefer “upgrade” strategies: read from existing DOM first, then enhance.
• Keep data “in the DOM” as much as practical, because it remains inspectable, archivable, diffable, and AI-friendly.
• When embedding XML, keep it inert (inside <template>), then parse it in the custom element.
• Avoid hidden client state that cannot be reconstructed from the HTML + config.
• Avoid DOM mutation that destroys the original meaning; if you replace markup, keep a reference or preserve semantics inside shadow DOM.

Suggested task breakdown:

1. Draft docs/hypermedia-object-models.md with the generalized pattern and conventions.
2. Extract reusable utilities from <natural-grid> into lib/hypermedia-ux/* without changing behavior.
3. Refactor <natural-grid> to use the shared utilities where it reduces duplication.
4. Build second component demonstrating the pattern.
5. Add minimal docs/examples for both components.
6. Add a small set of deterministic tests (string snapshots of generated DOM, config resolution order tests, plugin ordering tests).

The whole point is to make complex UI feel like “server-rendered HTML first,” while still allowing rich client behavior where needed, but only as progressive enhancement through custom elements. The <natural-grid> JSDoc describes most of the primitives we need. This ticket is about turning that into a generalized, repeatable pattern that can scale to many components without turning into a framework.