Skip to content

Commit de8d2c7

Browse files
dfa1claude
andcommitted
perf(writer): raise default global-dict retained budget 1GB -> 2GB (#303)
On a wide, high-cardinality file the per-column code buffers (~37 MB each, ADR 0021) sum past the 1 GB budget, and eviction demotes the largest-retained — i.e. the highest-cardinality — columns to per-chunk dictionaries, repeating their values pool every chunk. nyc-311 (~30 admitted string columns ≈ 1.15 GB retained) lost its three Cross Street / Street Name columns this way. Raise the default to 2 GB, which fits that file with headroom (measured: 2 GB and 3 GB both match the 8 GB result, so retained tops out ~1.15 GB) while still bounding the pathological many-wide-columns risk. Constrained-heap writers can still lower it via withGlobalDictMaxRetainedBytes(...). nyc-311 re-encode: 1934.23 MB -> 1879.34 MB (1.10x -> 1.07x vortex-jni). This overlaps #305 (both improve the street columns, via keeping-global vs bitpacked codes): stacked on #305 the budget adds ~35 MB net (1862 -> 1827 MB). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 8ab484c commit de8d2c7

2 files changed

Lines changed: 14 additions & 11 deletions

File tree

writer/src/main/java/io/github/dfa1/vortex/writer/WriteOptions.java

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -37,12 +37,15 @@ public record WriteOptions(
3737
boolean enableZstd,
3838
long globalDictMaxRetainedBytes
3939
) {
40-
/// Default aggregate retention budget (1 GB) for the buffered per-chunk code arrays of global
40+
/// Default aggregate retention budget (2 GB) for the buffered per-chunk code arrays of global
4141
/// -dictionary candidate columns. Raised from 256 MB when buffering became cardinality-bounded
42-
/// (ADR 0021): codes are ~35–45× smaller than the raw values the old budget guarded, so a 1 GB
43-
/// default bounds the same pathological many-wide-columns risk while letting normal wide,
44-
/// low-cardinality files (e.g. NYC 311, ~38 string columns) keep all their shared dictionaries.
45-
private static final long DEFAULT_GLOBAL_DICT_MAX_RETAINED_BYTES = 1024L * 1024 * 1024;
42+
/// (ADR 0021), then from 1 GB (#303): a wide, high-cardinality file (NYC 311, ~30 admitted string
43+
/// columns × ~37 MB of buffered codes ≈ 1.15 GB) crossed the 1 GB budget and evicted its
44+
/// highest-cardinality columns to per-chunk dictionaries, repeating their values pool each chunk
45+
/// (~35 MB larger). 2 GB fits that file with headroom while still bounding the pathological
46+
/// many-wide-columns risk. Constrained-heap writers can lower it via
47+
/// [#withGlobalDictMaxRetainedBytes(long)].
48+
private static final long DEFAULT_GLOBAL_DICT_MAX_RETAINED_BYTES = 2L * 1024 * 1024 * 1024;
4649

4750
/// Default options: global dictionary encoding enabled, no cascading compression, Zstd disabled.
4851
///

writer/src/test/java/io/github/dfa1/vortex/writer/WriteOptionsTest.java

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,14 +7,14 @@
77
/// Unit tests for [WriteOptions] factories and copy-methods.
88
class WriteOptionsTest {
99

10-
// The hardcoded default; both factories must keep supplying it so the budget stays 1 GB unless a
10+
// The hardcoded default; both factories must keep supplying it so the budget stays 2 GB unless a
1111
// caller overrides it via withGlobalDictMaxRetainedBytes(...). Raised from 256 MB when buffering
12-
// became cardinality-bounded (ADR 0021): the budget now guards ~2 B/row code arrays, not raw
13-
// values, so a larger budget bounds the same risk while keeping wide low-cardinality files dicted.
14-
private static final long DEFAULT_BUDGET = 1024L * 1024 * 1024;
12+
// became cardinality-bounded (ADR 0021), then from 1 GB (#303) so wide high-cardinality files
13+
// keep their high-cardinality columns globally dictionaried instead of evicting them to per-chunk.
14+
private static final long DEFAULT_BUDGET = 2L * 1024 * 1024 * 1024;
1515

1616
@Test
17-
void defaults_globalDictMaxRetainedBytes_is1Gb() {
17+
void defaults_globalDictMaxRetainedBytes_is2Gb() {
1818
// Given / When
1919
WriteOptions result = WriteOptions.defaults();
2020

@@ -23,7 +23,7 @@ void defaults_globalDictMaxRetainedBytes_is1Gb() {
2323
}
2424

2525
@Test
26-
void cascading_globalDictMaxRetainedBytes_is1Gb() {
26+
void cascading_globalDictMaxRetainedBytes_is2Gb() {
2727
// Given / When
2828
WriteOptions result = WriteOptions.cascading(3);
2929

0 commit comments

Comments
 (0)