Skip to content

Commit ca41a0f

Browse files
dfa1claude
andcommitted
test(parquet): measure imported Vortex file size vs source Parquet
Adds taxiParquet_importedSize_vsOriginal — imports NYC Yellow Taxi 2024-01 (~3M rows, 19 columns, mix of I64 / F64 / I32 / Utf8) via ParquetImporter and prints the size ratio. Uses the same cached parquet file as ParquetVsVortexReadBenchmark so no extra download in normal runs. Current measurement (after b4d1b43 added global dict for Utf8): Parquet = 49,961,641 bytes (47.6 MB) Vortex = 49,260,172 bytes (47.0 MB) ratio = 0.99x Vortex is now 1% smaller than Parquet on this dataset. The improvement over the prior 50 MB baseline tracks the new Utf8 global-dict path — taxi has several low-cardinality string-style columns whose per-chunk dicts were duplicated wastefully before. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent b4d1b43 commit ca41a0f

1 file changed

Lines changed: 32 additions & 0 deletions

File tree

integration/src/test/java/io/github/dfa1/vortex/integration/ParquetImportIntegrationTest.java

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,38 @@ void stringColumnValuesMatch(@TempDir Path tmp) throws Exception {
155155
}
156156
}
157157

158+
@Test
159+
void taxiParquet_importedSize_vsOriginal(@TempDir Path tmp) throws Exception {
160+
// Given — NYC Yellow Taxi 2024-01 (~3M rows, 19 cols, mix of I64 / F64 / I32 / Utf8).
161+
// Uses the same parquet file as ParquetVsVortexReadBenchmark (cached in /tmp).
162+
Path cached = Path.of("/tmp", "yellow_tripdata_2024-01.parquet");
163+
Path src;
164+
if (java.nio.file.Files.exists(cached)) {
165+
src = cached;
166+
} else {
167+
assumeNetworkAvailable();
168+
src = tmp.resolve("yellow_tripdata_2024-01.parquet");
169+
try (var in = URI.create("https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet")
170+
.toURL().openStream()) {
171+
java.nio.file.Files.copy(in, src);
172+
}
173+
}
174+
Path vortex = tmp.resolve("taxi.vortex");
175+
176+
// When
177+
ParquetImporter.importParquet(src, vortex);
178+
179+
// Then
180+
long parquetSize = java.nio.file.Files.size(src);
181+
long vortexSize = java.nio.file.Files.size(vortex);
182+
System.out.printf(
183+
"[TaxiSizeComparison] Parquet=%,d bytes (%.1f MB) Vortex=%,d bytes (%.1f MB) Vortex/Parquet=%.2fx%n",
184+
parquetSize, parquetSize / 1_048_576.0,
185+
vortexSize, vortexSize / 1_048_576.0,
186+
(double) vortexSize / parquetSize);
187+
assertThat(vortexSize).isGreaterThan(0);
188+
}
189+
158190
@Test
159191
void parquetAndVortexRowCountsAreEqual(@TempDir Path tmp) throws Exception {
160192
// Given

0 commit comments

Comments
 (0)