Commit de8d2c7
perf(writer): raise default global-dict retained budget 1GB -> 2GB (#303)
On a wide, high-cardinality file the per-column code buffers (~37 MB each,
ADR 0021) sum past the 1 GB budget, and eviction demotes the largest-retained —
i.e. the highest-cardinality — columns to per-chunk dictionaries, repeating their
values pool every chunk. nyc-311 (~30 admitted string columns ≈ 1.15 GB retained)
lost its three Cross Street / Street Name columns this way.
Raise the default to 2 GB, which fits that file with headroom (measured: 2 GB and
3 GB both match the 8 GB result, so retained tops out ~1.15 GB) while still bounding
the pathological many-wide-columns risk. Constrained-heap writers can still lower it
via withGlobalDictMaxRetainedBytes(...).
nyc-311 re-encode: 1934.23 MB -> 1879.34 MB (1.10x -> 1.07x vortex-jni). This
overlaps #305 (both improve the street columns, via keeping-global vs bitpacked
codes): stacked on #305 the budget adds ~35 MB net (1862 -> 1827 MB).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 8ab484c commit de8d2c7
2 files changed
Lines changed: 14 additions & 11 deletions
File tree
- writer/src
- main/java/io/github/dfa1/vortex/writer
- test/java/io/github/dfa1/vortex/writer
Lines changed: 8 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | | - | |
| 40 | + | |
41 | 41 | | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
46 | 49 | | |
47 | 50 | | |
48 | 51 | | |
| |||
Lines changed: 6 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
| 10 | + | |
11 | 11 | | |
12 | | - | |
13 | | - | |
14 | | - | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
0 commit comments