Scale context: the DuckLake deployment hoglake replaces stands at 2PB and ~1T rows. The straight answer to "how does hoglake improve on this": at this scale hoglake changes almost nothing about the data plane and nearly everything about the metadata plane — which is where every production scar came from. The parquet bytes are the same bytes. What changes is that the catalog stops being the bottleneck, the liability, and the incident generator.
- Storage cost / scan throughput of the data itself. 2PB of parquet is 2PB of parquet. Trino/Spark scan speed is the engine's problem and the file layout's problem — hoglake helps the layout only indirectly (compaction, stats quality).
- The object store. S3 throughput, listing costs, egress — unchanged.
| DuckLake at scale (observed) | Hoglake | How |
|---|---|---|
| Commit loads stats for the entire catalog: 5–7s/attempt at 59K tables / 3.7M stats rows; 190–264s commits under OCC retry | p50 2.0–3.6ms commits, flat vs. preseeded history (bench) | Write-set-scoped commit; O(catalog) loads structurally impossible. ~5 orders of magnitude on the pathological end |
| OCC conflict = snapshot-PK collision; retry re-pays the whole commit | Typed change rows, one indexed anti-join; appends never conflict with appends | hog_snapshot_change + (catalog, object, kind, snapshot) index; bench: k=8 writers, 368 commits/s, 0 conflicts |
| Expiry ~14ms/snapshot → ~50h to drain a 15.3M-snapshot backlog; DDL churn grows it forever | 5.6–14.5K snapshots/s (bench), continuous incremental sweeps | Range deletes + cascade; ~100–200× drain rate, continuous instead of cron-batched |
| 99.4% of stats rows for dropped tables; purging them bought 30–50× | Unrepresentable | FK ON DELETE CASCADE; drop/expiry removes stats rows with the file rows |
| 50M orphaned partition-value rows; 9.99M-row phantom deletion queue | Unrepresentable | FKs + drain-time liveness check; the queue is a suggestion, never an authorization |
| Compaction recovery emitting 60–180s commits → convoy, 13 restarts | max_groups_per_run (default 1) per sweep; commit is small metadata under the lock; execution entirely outside any transaction |
The 2026-09-04 incident shape designed out: foreground writers wait ms, never on S3 IO |
| Two engines, one catalog (millpond upstream + viaduck fork); drift; fork fixes never reaching maintenance | Exactly one catalog implementation, deployed once | Fork-drift / extension-distribution / OOM-loop class dies |
CDC: table_changes() read-barrier crawl; every consumer hand-builds cursors; expiry silently destroys unread ranges |
Changefeed API + catalog-resident consumer offsets + retention floor + 410 | hedgerow rows-then-offset, halt-on-incarnation; expiry pages when a consumer pins instead of silently losing its data |
| Rowid reuse on upsert-recreate; sorted compaction silently remaps rowids | Server-assigned, never-reused, explicit_row_ids through compaction |
Every CDC consumer's defensive machinery at 1T rows — retired |
| Recovery artisanal (split-brain reconstructed from S3 delete markers) | Liveness-checked soft deletion + forensics ledger + audit trail keyed to principals | The drain ledger keeps what was deleted, when, why, after how many attempts |
- Files: 2PB at ~256MB/file ≈ ~8M
hog_data_filerows. Trivial for Postgres with the partial live index. - Stats — the real volume driver: 8M files × ~50 columns ≈ ~400M
hog_file_column_statsrows, ~60–80GB with indexes. The catalog's dominant table by far. Postgres handles it (narrow PK-clustered btree, cascade-deleted with its file), but note it: the catalog stays small relative to 2PB (~0.004%), and stats are what you'd partition first if it ever stopped being small. The DuckLake version of this table was smaller only because it was broken (3.7M rows, 99.4% garbage). - Row-id allocator: 1T rows against a bigint
next_row_id— 2^63 headroom, a non-issue by design, with the CHECK +addExacttwo-layer defense against overflow. - Snapshots: the busiest writer mints ~138K/day; the advisory-lock tail sustains hundreds/s. ~100× headroom, and bench says commit latency doesn't move with history depth.
- One Postgres per service; one advisory-lock tail per catalog. Commit throughput per catalog is capped at 1/tail-latency. At the observed ~1.6 mints/s average (bursts higher), fine. If a single catalog ever needs sustained >100 commits/s, that's the wall. The escape hatch is to move validation in front of the serialization point rather than inside it: designed once, never scheduled, and no longer written down anywhere but here.
- Read scaling. Facade/Trino planning reads hit the service → one PG. Snapshot-pinned reads are replica-able (lag-tolerant by construction), but that's not built yet.
- Compaction IO cost at 2PB. The never-convoy design (one group
per sweep, full download+rewrite locally, heap-materialized
survivors) deliberately throttles compaction throughput. At 2PB
with continuous small-file ingestion, whether
max_groups_per_run=1keeps up with the small-file accumulation rate is a real capacity-planning question — the knob exists; the tradeoff is intentional. - Consumer-floor pinning at 2PB. A stuck consumer pins expiry → end-snapshotted files accumulate as storage. DuckLake's answer was silent loss; hoglake's is bounded staleness + a page. Correct tradeoff, but at 2PB the pinned-storage bill grows faster.
2PB doesn't move — the converter is metadata-only (paths carried, row-id allocators preserved, lineage spans the cutover). The real decision is snapshot-history depth: converting 15.3M snapshots is a choice, not a default — at this scale, head+retention-window. And the FK proof matters: every orphan class in the defect ledger is a row the new constraints will reject, so the converter is the audit of how dirty the old catalog actually is.
At 2PB/1T, DuckLake's failures were never about the data being big — they were about every commit, expiry, and maintenance run paying O(catalog) against a metadata store with no integrity and no arbiter. Hoglake's numbers at bench scale (flat commits, 0 conflicts, 100–200× expiry, still_referenced=0) are the structural answers to exactly those numbers. The data plane was never the problem.