Commit ca41a0f
test(parquet): measure imported Vortex file size vs source Parquet
Adds taxiParquet_importedSize_vsOriginal — imports NYC Yellow Taxi
2024-01 (~3M rows, 19 columns, mix of I64 / F64 / I32 / Utf8) via
ParquetImporter and prints the size ratio. Uses the same cached parquet
file as ParquetVsVortexReadBenchmark so no extra download in normal runs.
Current measurement (after b4d1b43 added global dict for Utf8):
Parquet = 49,961,641 bytes (47.6 MB)
Vortex = 49,260,172 bytes (47.0 MB)
ratio = 0.99x
Vortex is now 1% smaller than Parquet on this dataset. The improvement
over the prior 50 MB baseline tracks the new Utf8 global-dict path —
taxi has several low-cardinality string-style columns whose per-chunk
dicts were duplicated wastefully before.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>1 parent b4d1b43 commit ca41a0f
1 file changed
Lines changed: 32 additions & 0 deletions
File tree
- integration/src/test/java/io/github/dfa1/vortex/integration
Lines changed: 32 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
155 | 155 | | |
156 | 156 | | |
157 | 157 | | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
158 | 190 | | |
159 | 191 | | |
160 | 192 | | |
| |||
0 commit comments