docs(explanation): why-cascading walkthrough with measured numbers

Add a "Why cascading compression" section to docs/explanation.md
covering six representative columns (monotonic timestamps, low-card
categorical, random doubles, ALP-friendly doubles, RLE ints, high-card
strings). Each subsection shows the encoding tree the writer picks
with cascade off vs depth-3, ASCII diagrams, and the resulting size.

Numbers come from a new CompressionShowcase main class under
performance/. Runs 1 M rows per column and prints raw / no-cascade /
cascade-depth-3 file sizes — reproducible end-to-end.

Includes an "Encodings in play" glossary table so terms like
"ALP-friendly" and "FoR-friendly" are defined before the walkthrough.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Author: dfa1

docs/explanation.md

@@ -326,6 +326,310 @@ At decode time the registry maps the ID string from the Layout node to the right
 Custom encodings can be added at build time: `Registry.builder().register(myEncoding).build()`.
 Files with unrecognised IDs throw `VortexException` unless the builder enabled `allowUnknown()`.
 
+## Why cascading compression
+
+Vortex stores each column as a tree of encodings. The leaves are raw memory
+segments, the inner nodes describe how those bytes turn back into values.
+Without cascading, the writer picks one encoding from a static list and stops.
+With cascading, the writer samples the data, lets candidate encodings expose
+their open child slots, and recursively picks the best inner encoding for
+every child. That recursion is what turns a per-encoding sales-pitch into
+real compression.
+
+Six representative columns, both paths. Reproduce with:
+
+```
+./mvnw verify -DskipTests
+java --enable-native-access=ALL-UNNAMED -cp "..." \
+     io.github.dfa1.vortex.performance.CompressionShowcase
+```
+
+Numbers below: 1 000 000 rows per column, JDK 25, vortex-java HEAD.
+
+### The encodings in play, and what they're good for
+
+Before reading the table, you need to know what the writer's pick means. The
+labels in the size table aren't marketing terms — each one is a concrete
+encoding with a sweet-spot data shape.
+
+| Encoding         | What it does                                                                                           | "Friendly" data shape                                                |
+|------------------|--------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|
+| **Primitive**    | Raw little-endian bytes. The baseline.                                                                 | Truly random data — no overhead, no compression.                     |
+| **FoR**          | Frame-of-reference. Subtract `min(col)` from every value; store the minimum + the residuals.           | Bounded range: timestamps, monotonic IDs, anything where `max - min` is small. |
+| **Bitpacked**    | Packs N values into the smallest bit-width that fits the maximum value.                                | Small-range integers (e.g. FoR residuals, dict codes, small counts). |
+| **Dict**         | Build a values table; store one code per row pointing into it.                                         | Low-cardinality strings or numbers. **Loses** above ~50 % distinct.  |
+| **ALP**          | Adaptive Lossless float compression. Detects a per-column scale + exponent so that values become small integers, then stores the mantissa. | "Physical world" doubles — prices, sensor readings, percentages. **Loses** on truly random F64. |
+| **RLE / RunEnd** | Replace each run of identical values with `(run_end_index, value)`.                                    | Long runs of repeated values — status flags, partition IDs, slow-tick counters. |
+| **VarBin**       | Concatenate all bytes, store offsets per row. Arrow's classic string layout.                           | Strings of any cardinality — the safe default for Utf8.              |
+| **FSST**         | Builds a per-column symbol table of the most common byte bigrams, then rewrites the strings as 1-byte codes (or escape + literal). | Short, repetitive-ish strings — log lines, identifiers, JSON keys. Truly random strings beat it. |
+| **Constant**     | Store the scalar once. Done.                                                                           | All values identical.                                                |
+
+So "ALP-friendly doubles" = doubles that can be written as `mantissa × 10^-exp`
+with a small mantissa. "FoR-friendly" = bounded range. "RLE-friendly" = long
+runs. The cascading compressor does the matchmaking; this table tells you
+when each match is a win.
+
+### Headline table
+
+```
+dataset                          raw bytes    no-cascade   cascade(3)    ratio
+--------------------------------------------------------------------------------
+monotonic-timestamps              8 000 000   8 000 664    2 501 832     3.20x
+low-card-categorical              4 200 000   1 000 921    1 000 921     4.20x
+random-doubles                    8 000 000  12 751 764    8 000 672     1.00x
+alp-friendly-doubles              8 000 000   8 000 748    1 256 208     6.37x
+rle-int                           4 000 000   4 000 656        1 604  2 493.77x
+highcard-strings                  6 000 000  17 977 750   10 551 040     0.57x
+```
+
+ratio = raw / cascade(3).
+
+Three patterns to notice before we dig in:
+
+- **Cascading is usually a strict improvement** — never bigger, often dramatically
+  smaller (RLE: 4 MB → 1.6 KB; ALP-friendly F64: 8 MB → 1.3 MB).
+- **Without cascading, the writer can make the file *bigger* than raw.** Static
+  fallbacks pick the first encoding that "accepts" the dtype, even when it
+  fails to compress (random F64 → ALP adds overhead; high-cardinality strings
+  → Dict blows up the values vector).
+- **The cascade isn't magic.** Truly random data ends up close to raw size
+  because there's no structure to exploit.
+
+### 1. Monotonic timestamps — FoR + Bitpacked
+
+Sensor / log streams produce strictly increasing UNIX timestamps. Each value
+is only ~1 second above the previous one, so the *delta* needs ~1 bit, but
+the *absolute* value is 64 bits.
+
+```
+dtype: I64
+sample data: [1767225600, 1767225601, 1767225602, 1767225603, …]
+```
+
+**Without cascading** — `Primitive` (raw 8 bytes/row) → 8 MB.
+
+```
+                 Primitive(I64)
+                       │
+                  8 000 000 bytes
+```
+
+**With cascading depth 3** — sample shows all-positive small deltas. FoR
+subtracts the minimum so residuals fit in a tiny number of bits; the open
+residual child cascades into Bitpacked, which packs `~20` bits per row
+instead of 64.
+
+```
+                       FoR
+              ref = 1767225600
+                       │
+                  Bitpacked
+                bit_width = 20
+                       │
+              ~2.5 MB packed
+```
+
+3.2× smaller than raw. The same shape covers row IDs, monotonic counters,
+millisecond timestamps, anything with bounded local deltas.
+
+### 2. Low-cardinality categorical — Dict
+
+A column of ticker symbols repeats the same 5 distinct strings across a
+million rows.
+
+```
+dtype: Utf8
+sample data: ["AAPL", "MSFT", "NVDA", "GOOGL", "AMZN", "AAPL", "MSFT", …]
+```
+
+**Without cascading** — the default codec list picks Dict before VarBin, so
+even the no-cascade path catches this one.
+
+```
+                       Dict
+                   /         \
+              Values         Codes
+            (5 strings)   (1 M × U8)
+              ~20 B          ~1 MB
+```
+
+**With cascading depth 3** — same shape. Dict already wins; codes are tight
+(1 byte each because cardinality ≤ 256). 4.2× smaller than raw with no
+recursive work needed.
+
+The new cardinality gate (added in 0.6) only kicks in *above* 50 % distinct;
+this column is at 0.0005 % distinct, far below the gate.
+
+### 3. Random doubles — Primitive (no compression possible)
+
+The worst case. Truly random F64 values have no exploitable structure.
+
+```
+dtype: F64
+sample data: [0.733, 0.642, 0.218, 0.875, 0.157, 0.488, 0.999, …]
+```
+
+**Without cascading** — the static list tries ALP first. ALP detects no
+common scale factor, falls back to its uncompressed path, and the resulting
+file is *bigger* than raw (12.7 MB vs 8 MB) thanks to per-row mantissa /
+exponent bookkeeping.
+
+```
+                  ALP (degenerate)
+                /        \    \
+            Encoded   Patch    Patch
+            (no-op)   idx      values
+                      8 MB     ~4.7 MB
+```
+
+**With cascading depth 3** — the cost-based selector measures ALP, sees it's
+worse than primitive, and picks `Primitive` raw.
+
+```
+                Primitive(F64)
+                      │
+                  8 000 000 bytes
+```
+
+Lesson: cascading is the cheapest insurance against the writer making the
+file *larger*. Without it, "first match" can lose to raw bytes.
+
+### 4. Slowly-varying doubles — ALP + Bitpacked
+
+Stock prices, sensor readings, and most "physical world" doubles drift
+slowly. They're representable as `mantissa × 10^-exp` with a small mantissa,
+which is exactly what ALP is for.
+
+```
+dtype: F64
+sample data: [100.05, 100.04, 100.06, 100.07, 100.05, 100.03, …]
+```
+
+**Without cascading** — ALP picks the right shape, but its mantissa child
+gets emitted as raw `Primitive(I64)` → ~8 MB.
+
+```
+                       ALP
+                     e=2 f=1
+                       │
+                  Primitive(I64)
+                       │
+                  8 000 000 bytes
+```
+
+**With cascading depth 3** — the same ALP outer, but its mantissa child
+cascades into FoR + Bitpacked. The mantissa range fits in ~10 bits.
+
+```
+                       ALP
+                     e=2 f=1
+                       │
+                       FoR
+                       │
+                  Bitpacked
+                bit_width = 10
+                       │
+                  ~1.25 MB
+```
+
+6.4× smaller than raw. This is where Vortex really shines vs Parquet's
+fixed page-level codecs: nested arithmetic encodings stack cleanly.
+
+### 5. Run-encoded ints — RunEnd
+
+Long runs of the same value: status flags, monotonic counters that tick
+slowly, partition IDs.
+
+```
+dtype: I32
+sample data: [1,1,1,…(10 000)…,1, 2,2,2,…(10 000)…,2, 3,3,…]
+```
+
+**Without cascading** — `Primitive` → 4 MB.
+
+```
+                 Primitive(I32)
+                       │
+                  4 000 000 bytes
+```
+
+**With cascading depth 3** — the run structure is detected, RunEnd encodes
+each run as `(end_index, value)` and both children compress.
+
+```
+                     RunEnd
+                   /        \
+             Run ends     Values
+             (100 ints)   (100 ints)
+              ~400 B       ~400 B
+                  ↓             ↓
+              Bitpacked     Bitpacked
+              (or FoR)       (or FoR)
+```
+
+**2 493×** smaller than raw — the run is so long that the actual payload
+collapses to ~1.6 KB.
+
+### 6. High-cardinality strings — Dict fail / FSST partial win
+
+A million all-distinct random 6-character strings — the kind of column that
+turns into a tar pit for dictionary-style encodings.
+
+```
+dtype: Utf8
+sample data: ["wkzqof", "tdmgxh", "ablrpe", "yvcjsi", …]
+```
+
+**Without cascading** — Dict is the first acceptor for Utf8 in the default
+codec list. It builds a dictionary nearly as big as the input plus a 4-byte
+code per row.
+
+```
+                       Dict
+                   /         \
+              Values         Codes
+            (1 M strings)  (1 M × U32)
+              ~10 MB         ~4 MB
+                                =  18 MB (3× raw!)
+```
+
+**With cascading depth 3** — the new (0.6) cardinality gate in `DictEncoding`
+detects > 50 % distinct on the sample, returns `notApplicable`, and the
+cascade rotates to FSST.
+
+```
+                       FSST
+              ╭────────┼────────╮
+          Symbol     Symbol    Compressed
+           table     lengths    payload
+           (~2 KB)    (~255 B)   (~5 MB)
+                       │
+                   ↓ cascade
+              uncompressed_lens     codes_offsets
+              (Constant: ~4 B)      (FoR+Bitpacked)
+                                       ~5 MB
+```
+
+Result: 10.5 MB — still larger than the 6 MB raw, but **42 % smaller than
+no-cascade**. Truly random short strings are hard for any encoder (Rust hits
+the same wall on this input). Java's FSST symbol-table builder is also less
+aggressive than Rust's for now — see `TODO.md`.
+
+### Takeaways
+
+- **Always use `WriteOptions.cascading(...)` unless you have a reason not
+  to.** The default is `cascading(0)` for legacy compatibility; we'll
+  probably flip the default in a future major.
+- **Cascade depth 3** is the sweet spot in practice. Deeper costs more
+  encode CPU for tiny diminishing returns; shallower misses key combos
+  like ALP→FoR→Bitpacked.
+- **The encoding tree is the API.** If you `vortex inspect <file>` you'll
+  see the exact structure of each column, with sizes per node. No black
+  box.
+- **Codec choice is data-dependent.** No single encoding is "best". The
+  point of cascading is to let the writer admit when it's wrong and try
+  again.
+
 ## Benchmarks
 
 JMH throughput (ops/s = full-file scans per second). Higher is better.