Skip to content

Commit 6e9f139

Browse files
dfa1claude
andcommitted
docs: refresh benchmark tables with current measurements (2026-06-08)
Old numbers (127.9 / 235.1 / 122.6) drifted from reality by 20-80%; bisecting that drift consumed real time in the perf session. Replace with honest current snapshot against commit 051a794, dated, with JMH config recorded inline so future drift is locatable. Changes: - OHLC read: replace doc baseline with current values close 76.7 → 61.0 ± 5.8 volume 127.9 → 104.8 ± 5.1 (recovered from pre-fix 83 via broadcast fast-path) symbol 110.4 → 97.8 ± 1.8 + cascading (depth 3): 80.9 ± 1.2 (new row, was already in the benchmark) - Parquet vs Vortex taxi: re-measure parquetRead 166.5 → 137.0 (stable) vortexRead 235.1 → 43.0 (regressed — flagged inline; bisect tracked separately) parquetReadMultiColumn 133.0 → 137.4 (stable) vortexReadMultiColumn 122.6 → 34.1 (regressed) - Tone down "Why Vortex is faster" narrative to "format-level advantages (theory)" until the Parquet-vs-Vortex regression is fixed. Properties still hold; the measurement just doesn't currently support the claim. - OHLC write + Big-file scan: mark as "last measured before 2026-06-08; re-run pending" since not re-measured today. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 454d3ba commit 6e9f139

1 file changed

Lines changed: 35 additions & 25 deletions

File tree

docs/explanation.md

Lines changed: 35 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -328,17 +328,19 @@ Files with unrecognised IDs throw `VortexException` unless the builder enabled `
328328

329329
## Benchmarks
330330

331-
JMH throughput (ops/s = full-file scans per second). Higher is better.
331+
JMH throughput (ops/s = full-file scans per second). Higher is better. Numbers
332+
re-measured 2026-06-08 against commit `051a794`.
332333

333-
**Environment:** Apple M5, OpenJDK 25, 3 warmup × 3 s, 5 measurement × 5 s, fork 1.
334+
**Environment:** Apple M5, OpenJDK 25, 5 warmup × 3 s, 10 measurement × 5 s, fork 1.
334335

335336
### OHLC read — 10 M rows, 58.9 MB (Rust-written file, single-column projection)
336337

337-
| Benchmark | Java (ops/s) | JNI/Rust (ops/s) | Java speedup |
338-
|-----------------|--------------|------------------|--------------|
339-
| close (F64/ALP) | 76.7 ± 0.3 | 50.4 ± 2.8 | **1.5×** |
340-
| volume (I64) | 127.9 ± 2.3 | 52.9 ± 0.6 | **2.4×** |
341-
| symbol (varbin) | 110.4 ± 0.4 | 9.6 ± 0.9 | **11.5×** |
338+
| Benchmark | Java (ops/s) | JNI/Rust (ops/s) | Java speedup |
339+
|---------------------|---------------|------------------|--------------|
340+
| close (F64/ALP) | 61.0 ± 5.8 | 47.9 ± 0.7 | **1.3×** |
341+
| volume (I64/bitpacked) | 104.8 ± 5.1 | 48.4 ± 1.7 | **2.2×** |
342+
| symbol (varbin) | 97.8 ± 1.8 | 9.2 ± 0.4 | **10.6×** |
343+
| cascading (depth 3, volume) | 80.9 ± 1.2 | n/a ||
342344

343345
### OHLC write — 10 M rows
344346

@@ -347,20 +349,23 @@ JMH throughput (ops/s = full-file scans per second). Higher is better.
347349
| write | 4.4 ± 1.1 | 0.7 ± 0.1 | **6.4×** |
348350

349351
The Java write is faster but also produces bigger files (more optimization work remains).
352+
_Last measured before 2026-06-08; re-run pending._
350353

351354
### Big-file scan — 100 M rows × 4 I64 columns, ~3 GB (Rust-written file, all columns)
352355

353356
| Benchmark | Java (ops/s) | JNI/Rust (ops/s) | Java speedup |
354357
|-----------|--------------|------------------|--------------|
355358
| scan | 20.4 ± 0.9 | 5.7 ± 0.6 | **3.6×** |
356359

360+
_Last measured before 2026-06-08; re-run pending._
361+
357362
### Parquet vs Vortex read — NYC Yellow Taxi 2024-01, 3 M rows, 19 columns
358363

359364
Both formats store all 19 columns; projection happens at read time. Both sides scalar decode
360365
(Hardwood disables SIMD on JDK 25; Vortex Java uses FFM scalar reads throughout).
361-
File sizes: Parquet 47.6 MB, Vortex Java 50 MB.
362366

363-
**Environment:** Apple M5, OpenJDK 25, 5 warmup × 3 s, 5 measurement × 5 s, fork 2.
367+
**Environment:** Apple M5, OpenJDK 25, 5 warmup × 3 s, 10 measurement × 5 s, fork 1.
368+
Re-measured 2026-06-08 against commit `051a794`.
364369

365370
Two Parquet variants are measured to isolate format cost from API overhead:
366371

@@ -371,22 +376,27 @@ Two Parquet variants are measured to isolate format cost from API overhead:
371376

372377
| Benchmark | ops/s | vs Parquet batch |
373378
|--------------------------------------------------------------------------|--------------|--------------------------|
374-
| `parquetRead` — batch, 1 col (`trip_distance`) | 166.5 ± 4.0 | baseline |
375-
| `parquetReadRowByRow` — row cursor, 1 col | 67.6 ± 4.4 | 0.41× (2.5× API penalty) |
376-
| `vortexRead` — 1 col (`trip_distance`) | 235.1 ± 6.9 | **1.41×** |
377-
| `parquetReadMultiColumn` — batch, 2 cols (`fare_amount`, `PULocationID`) | 133.0 ± 18.3 | baseline |
378-
| `parquetReadMultiColumnRowByRow` — row cursor, 2 cols | 44.0 ± 2.2 | 0.33× (3× API penalty) |
379-
| `vortexReadMultiColumn` — 2 cols | 122.6 ± 3.3 | 0.92× |
380-
381-
Single-column: Vortex 1.4× faster than Parquet batch — format advantage is real (mmap
382-
zero-copy + ALP vs Parquet RLE/ZSTD page decode).
383-
384-
Multi-column: Vortex (122.6) slightly behind Parquet batch (133.0). Known gap: Rust uses a
385-
global dict per column (one tiny dict for all 3 M rows); Java applies dict per 131 K-row
386-
chunk, increasing per-chunk overhead for low-cardinality columns like `PULocationID` (260
387-
unique values).
388-
389-
#### Why Vortex is faster on single-column reads
379+
| `parquetRead` — batch, 1 col (`trip_distance`) | 137.0 ± 14.8 | baseline |
380+
| `parquetReadRowByRow` — row cursor, 1 col | 69.7 ± 0.9 | 0.51× (2× API penalty) |
381+
| `vortexRead` — 1 col (`trip_distance`) | 43.0 ± 1.5 | **0.31×** |
382+
| `parquetReadMultiColumn` — batch, 2 cols (`fare_amount`, `PULocationID`) | 137.4 ± 10.7 | baseline |
383+
| `parquetReadMultiColumnRowByRow` — row cursor, 2 cols | 40.7 ± 1.9 | 0.30× (3.4× API penalty) |
384+
| `vortexReadMultiColumn` — 2 cols | 34.1 ± 1.6 | 0.25× |
385+
386+
**Known regression vs 2026-06-05 snapshot** (`vortexRead` was 235 → 43; `vortexReadMultiColumn`
387+
was 122 → 34, Parquet path stable). The collapse is in the Vortex decode path on the
388+
`ParquetImporter`-generated file — likely a cascade choice change that landed between
389+
`363a885` and `051a794`. The OHLC bench (raw I64/F64 columns) recovered to 100+ ops/s
390+
with the broadcast fast-path fix; this one did not, which points at a path the broadcast
391+
fix doesn't cover (probably dict-of-ALP or ZSTD-on-F64 sneaking into the cascade). Bisect
392+
+ fix tracked separately — these numbers are the current honest snapshot, not the target.
393+
394+
#### Format-level advantages (theory)
395+
396+
The bullets below describe the structural reasons Vortex *should* outperform Parquet on
397+
single-column reads, and did so in the 2026-06-05 measurement (235 → vs Parquet's 166).
398+
The current Vortex score sits below Parquet on this benchmark while the regression noted
399+
above is being investigated; the format properties themselves are unchanged.
390400

391401
**0. O(1) random access within a column.**
392402
Fixed-width encodings (ALP, BitPacked) make row N directly addressable:

0 commit comments

Comments
 (0)