Commit 0e3d35c
fix(writer): train ALP-RD dictionary on a stratified sample
ALP-RD built its left-parts dictionary from the first SAMPLE_SIZE physical rows
(findBestDictionaryF64/F32 iterated values[0..sampleLen)). The cascade measures
ALP-RD's cost on its own stratified sample, so a head-only dictionary could look
cheap in the competition yet flood the tail with exceptions on the full re-encode
when the leading rows are unrepresentative (sorted or clustered floats) — a size
regression newly reachable now that ALP-RD is a top-level cascade candidate (#304
review). Train on a stratified sample spanning the whole array instead, matching
AlpEncodingEncoder.findExponentsF64.
Also harden AlpRdCascadeSelectionIntegrationTest: besides asserting the categorical
ALP-RD win (decided on a 4096-row sample), assert the file compresses below raw F64,
so the test can't pass on a degenerate "selected but didn't beat raw" outcome.
nyc-311 unchanged at 1644.25 MB (its leading rows were already representative);
408 round-trip property tests, 16 ALP-RD unit tests, and 217 interop tests pass;
full verify green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 4228c30 commit 0e3d35c
2 files changed
Lines changed: 27 additions & 2 deletions
File tree
- integration/src/test/java/io/github/dfa1/vortex/integration
- writer/src/main/java/io/github/dfa1/vortex/writer/encode
Lines changed: 8 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
| |||
58 | 59 | | |
59 | 60 | | |
60 | 61 | | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
61 | 69 | | |
62 | 70 | | |
63 | 71 | | |
| |||
Lines changed: 19 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
52 | | - | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
53 | 63 | | |
54 | 64 | | |
55 | 65 | | |
| |||
128 | 138 | | |
129 | 139 | | |
130 | 140 | | |
131 | | - | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
132 | 149 | | |
133 | 150 | | |
134 | 151 | | |
| |||
0 commit comments