Schema centric view#47

Draft

grantjllnl wants to merge 26 commits intomainfrom

schema-centric-view

Commits on Oct 14, 2025

WIP - JSON Schema to py dataclass and ddl
Grant Johnson
committed

Commits on Oct 17, 2025

WIP - Added README, still working on SQL
Grant Johnson
committed

Commits on Nov 5, 2025

WIP - generates a table for each metadata type
Grant Johnson
committed
WIP - Schema with duckdb hints.
Grant Johnson
committed

Commits on Nov 6, 2025

Fixed PK datatypes, added support for single, wide table for subtypes with views atop
Grant Johnson
committed
Schema file, with a few manual tweaks, loader for testing
Grant Johnson
committed
Seths latest schema without duckdb annotations
Grant Johnson
committed
Merge branch 'main' into schema-centric-view
Grant Johnson
committed
Document filetype to metadata mapping
Grant Johnson
committed

Commits on Nov 7, 2025

View for base_metadata table. Enumerates all fields in structure
Grant Johnson
committed
Documentation for x-duckdb directives
Grant Johnson
committed
Working versions. Note that these have some manual edits.
Grant Johnson
committed
WIP - Add type override, allows BIGINT, fixed filetype_type name problem, added support for SEQ for PK. Still need to fix invalid column name: 'cert._version'
Grant Johnson
committed

Commits on Nov 24, 2025

Merged in latest observation.schema.json and added duckdb directives.
Grant Johnson
committed

Commits on Dec 18, 2025

Misc changes. Note, SQL first method is deprecated. DLT wins!
Grant Johnson
committed
WIP - Initial DLT loader
Grant Johnson
committed

Commits on Jan 3, 2026

WIP - Columns working, now to fix tables
Grant Johnson
committed

Commits on Jan 4, 2026

WIP - Added table processing. Kinda works.
Grant Johnson
committed

Commits on Jan 5, 2026

WIP - Generates Schema, Table, Column using filesystem source
Grant Johnson
committed

Commits on Jan 20, 2026

Simple Raw Data Viewer
Grant Johnson
committed

Commits on Mar 4, 2026

Loads surfactant data into duckdb. Used for debugging and initial data analysis
Grant Johnson
committed
Moved functionality
Grant Johnson
committed
Initial DLT loader
Grant Johnson
committed

Commits on Mar 5, 2026

DLT schema helpers
Grant Johnson
committed
Streamlit App
Grant Johnson
committed
Intro of bronze/silver schemas.
Grant Johnson
committed