Repository navigation
Conversation
Replace the legacy PAR2 runtime with bounded pure Julia metadata, schema, encoding, page, logical, nested reader, and writer layers. Add N5 and N6 conformance evidence, resource and mutation tests, documentation, CI, and the continuation handoff. Keep the package at preproduction status. Publication and oracle locking remain disabled until the remaining provenance and platform gates pass. BREAKING CHANGE: remove the legacy PAR2, cursor, reader, writer, and dataset implementation in favor of the new namespaced File, Table, and write API.
Adversarial review of the rewrite surfaced defects in metadata handling,
temporal round trips, and resource accounting. Fix them, then delete the
read and write paths that review proved unreachable.
Untrusted metadata
- Store preserved Thrift fields as `Vector{RawField}` instead of a tuple.
The tuple length was attacker controlled, and the generated `==`,
`isequal`, and `hash` specialize per length, so a small footer carrying
a few thousand unknown fields cost seconds of compilation on the first
comparison of decoded metadata. Re-check the union invariant when
encoding, because the vector is mutable after construction.
- Bound `max_schema_name_bytes` per operation instead of over the process
lifetime. One hostile file used to exhaust the shared budget and deny
every later file that carried a new column name.
- Reject a nested plan deeper than the reader's recursion limit with a
`LimitError`. The schema parser and plan compiler are iterative, so a
raised `max_metadata_depth` could otherwise exhaust the stack during
assembly and corrupt process state.
- Charge byte-stream-split fixed-width copies to the live budget, and
make the close guards atomic so a concurrent close cannot release a
materialization charge twice.
Temporal and rewrite fidelity
- Read a millisecond TIMESTAMP as `Timestamp{:millis}` when it is
adjusted to UTC. It previously decoded to a bare `DateTime`, so a read
and write cycle silently republished a UTC instant as local time.
Millisecond columns without adjustment still read as `DateTime`, and
the constructor now rejects the contradictory combination.
- Preserve `key_value_metadata` on a schema-bearing rewrite, with the
key and value sizes checked before allocation. Rewrites previously
dropped producer metadata such as `ARROW:schema`.
- Compare preserved fields exactly during provenance validation, so a
change to the raw Compact Thrift header state cannot pass as identical.
- Report an INT96 column as an unsupported feature rather than an invalid
file. INT96 is valid but deprecated, and is not implemented here.
Unreachable code
- Delete the flat LIST assembler in the table reader. Every schema
containing a group compiles to a nested plan, so the branch was
unreachable from `Parquet.Table`.
- Delete the MAP iterate fallbacks in the nested writer. A MAP value is
always a concrete `MapValue` or `AbstractDict`, both intercepted
earlier.
- Delete the unused column page builders and the unbudgeted dictionary
helpers they were the last callers of.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document the behavior that review found surprising but correct, and stop resolved environments from reaching the branch. - Note which annotations a read and write cycle does not preserve, and point at `LogicalColumn` for selecting an exact annotation. - State the reader's nesting depth bound and why the writer accepts deeper synthetic values than the reader will assemble. - Describe memory-mapped file lifetime: `close` releases the descriptor while the mapping waits for finalization, which can hold a Windows file open, and truncation by another process is fatal to the mapping. - Say what `max_materialized_bytes` covers, since borrowed vectors and mapped paths are not charged and an external source owns its storage. - Ignore the root and docs manifests and the corpus checkout. The conformance manifest under test/conformance/n6/julia stays tracked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The N6 evidence binds each producer to an exact source revision, so the preceding source changes invalidated it. Re-establish that binding by re-running the producers and the exact external gate, rather than by restating hashes. - Recompute the Parquet.jl producer descriptor over the new source using the gate's own composite algorithm. - Rebind the parquet-jl authority in the capability matrix. The producer harness refuses to emit evidence until the declared revision and toolchain match the descriptor. - Regenerate the producer evidence. The twelve generated Parquet files and ninety-five of the ninety-six evidence records are unchanged; only the authority record moved, which confirms the source changes leave the observable behavior identical. - Restate the capability digest each normalized evidence run record carries, and resolve the upstream evidence citations that follow from it. - Re-pin the frozen evidence digests and the artifact manifest. Verified with the exact external gate on macOS 15 arm64 against the pinned toolchain: 5117 of 5117 preproduction checks and 578 of 578 independent model checks, with the raw Java scanner, Arrow Rust, Parquet Java, PyArrow, and DuckDB oracles and the isolated schema validator. Gate status stays preproduction; publication and oracle locking remain unauthorized. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First continuous integration run of this branch exposed portability defects that only appear away from the development platform. - Install dependencies in the bounds job. It ran the N5 conformance suite without a build step, so the package failed to load. - Keep text files LF in the working tree. Windows checked out CRLF, which broke a multiline pattern in the N5 model tests. Conformance evidence and artifact manifests are content hashed, so a CRLF checkout would also change their digests. Binary fixtures are excluded, and no tracked file is renormalized by this change. - Compare a manifest path in its own forward-slash form. normpath emits the platform separator, so Windows rejected a correctly normalized path. - Loosen the writer scratch-stack allocation bounds. The same measurement runs about a seventh higher on x86-64 than on arm64, and the previous bound was tuned on arm64. Measured against the branch point on one machine the allocations are unchanged, so this is platform headroom rather than a regression. The bounds still catch a pass that stops reusing its scratch stacks, which costs far more than a constant per row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #199 +/- ##
==========================================
+ Coverage 86.84% 91.45% +4.60%
==========================================
Files 10 49 +39
Lines 1604 15822 +14218
==========================================
+ Hits 1393 14470 +13077
- Misses 211 1352 +1141 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
`close!` releases the descriptor while the mapping waits for finalization, so Windows keeps the file locked and removing the temporary file is refused. That is the documented source lifetime, so let this cleanup step accept the refusal instead of failing the suite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`relpath` emits the platform separator, so the harness looked for a backslash path in a manifest that records forward slashes and reported the included N6 model source as unpinned on Windows. The harness is itself pinned, so re-freeze the descriptor, the capability authority that names it, the producer evidence, the evidence bindings that cite it, and the artifact manifest. The source composite is unchanged, because the harness is a support file rather than package source. Re-verified with the exact external gate on macOS 15 arm64: 5117 of 5117 preproduction checks and 578 of 578 independent model checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The harness fsyncs its destination directory and relies on POSIX rename and symlink behavior, and refuses to publish anywhere else. The guard tests called it unconditionally, so Windows reported a refusal it was always going to make. Only the artifact manifest and its digest move; the producer descriptor, capability authority, and evidence are untouched, because this file is pinned as a static artifact rather than as producer source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This is very cool, but you should make a new package for this, because this PR is too big for people to review. |
Haha, I'm not sure a Parquet4.jl would go over well! No, I need to dig in to the diff here, because it's actually a way smaller set of actual code changes (comparable to the ARrow.jl package) and not actually 100K lines of julia to support parquet. I've only reviewed about half of it myself so far, so there's still probably ways to clean it up further. My understanding is that Parquet/Parquet2/Parquet3 are have various warts/gaps in spec coverage and none are super actively maintained. I'd like to have a really good package that does cover teh spec and can be pointed at as the "canonical" package; Parquet.jl seems like the best home for that. This isn't quite ready yet, but I'd love ot help anyone interested in helping digest a large codebase inot manageable chunks. |
|
Parquet2.jl is actively maintained and, right now I would call QuackIO.jl the canonical package. Parquet is hard because if you take this format seriously you should engage with upstream to ensure changes that are made to the format are compatible with the julia package. For example it would be good to see the hypothetical canonical julia package on https://parquet.apache.org/docs/file-format/implementationstatus/ to see how it stacks up against the standard implementations. Also as you can see from there, that nobody wants to have full spec coverage (there are deprecated things like LZO and one of the LZ4 codecs that you probably don't want to deal with). The spec is also vague and evolving over time, so in reality the target should be compatibility with Parquet2.jl, QuackIO.jl and the implementations in other languages. |
|
Also, is it possible to contribute improvements to https://github.com/JuliaAPlavin/QuackIO.jl/pulls and https://gitlab.com/ExpandingMan/Parquet2.jl/-/merge_requests instead of making another package? Maybe @aplavin @ExpandingMan or @bjarthur have some thoughts on this. |
|
I'm generally using QuackIO (admit I can be biased :) ), and would say its approach has both pros and cons. Obvious |
Is it? Last I heard, ExpandingMan was specifically not going to do further work on the package and was effectively "done" with it. I guess there are still dependabot bumps going on, but in my mind, it's not an actively developed package much more than Parqet.jl is. I have no issues/qualms with QuackIO.jl and I think it's an excellent option/format for certain use-cases. But it's also not what I'm personally interested in, which is pure-Julia parquet support. I'm also aware of the deprecated parts of the spec and don't plan to support them. Not trying to rock anyone's boat here or anything; I happen to have access to a lot of AI tokens at the moment, I have a lot of experience w/ data format implementations, and I've wanted a solid pure-Julia parquet for a long time with good mmap support, all hte Tables.jl integrations (including a new Tables.Scan component), etc. |
A MAP key snapshot exists only while a trace records one, so an aggregate scan run without a trace reached the key assertion with nothing to assert against and raised a MethodError. Struct and list sources already scanned without a trace; only MAP could not. Add the missing method for the absent snapshot. It keeps every check that does not need one, so only the comparison against a previous pass is skipped, and that comparison has no referent without a trace. The public writer always supplies a trace and never reached this. The regression covers both MAP scan arms, a dictionary source and a map view, and asserts that the scan still infers decimal and timestamp leaf parameters rather than merely completing. Re-freezing the N6 evidence follows from touching pinned package source. The twelve generated fixtures are unchanged. Re-verified with the exact external gate on macOS 15 arm64: 5117 of 5117 preproduction checks and 578 of 578 independent model checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The metadata types come from `julia thrift/generate.jl` and the N6 evidence records from the producer harness. Neither is edited by hand, so collapse both by default in review rather than reading them as authored code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Yes the Parquet2.jl package is very much alive. ExpandingMan is not driving development of new features, but last I checked he is still interested in supporting more Parquet features and willing to merge things and help maintain the code, your new way of handling thrift and all the java tests here seem like things that would be great in Parquet2.jl as well. From my perspective the only reason to keep using Parquet.jl is for people who like its API and don't want to learn Parquet2.jl or QuackIO.jl's API, so I can't see the reason to make breaking changes to the API of Parquet.jl IMO mmap often causes more problems then it is worth, and it is possible to match performance and have better error handling with manual caching systems (but I guess this depends on what operating system/file system/CPU you are using). |
|
Parquet2.jl is maintained in the sense that I intend to merge PR's and fix critical bugs, but yes it is very unlikely I will ever do more work than that on the package. I do think duckdb is a really good solution for parquet in Julia and it supports some fantastic features such as querying. In any case, I don't personally have any objections to Parquet2.jl being replaced with something incompatible. |
Complete coverage of the format was never a realistic goal, and treating it as one held the release behind work nobody has asked for. Decide these two out. LZO is stable and nondeprecated in the IDL, but every available implementation is GPL-2, so an MIT core cannot carry one. It was blocking the format-completeness claim behind a license-compatible implementation that does not exist. INT96 is deprecated in the format and was never implemented, although the roadmap claimed both directions. Neither makes a file invalid, so both now report an unsupported feature rather than an invalid file, matching how every other recognized-but-unsupported feature is reported. Record both as decided rather than pending, and state plainly that complete coverage is not a goal. The deprecated LZ4 codec and the deprecated BIT_PACKED encoding stay readable and stay unwritable. Files in the wild use them, so dropping the read path would cost compatibility rather than buy simplicity. Re-freezing the N6 evidence follows from touching pinned package source. Verified with the exact external gate on macOS 15 arm64: 5117 of 5117 preproduction checks and 578 of 578 independent model checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update the README and guide to match the recursive nested reader and writer, metadata preservation, and current page-index behavior. Add a runnable public nested example and reconcile the handoff with the existing preproduction PR. AI disclosure: This work was prepared with assistance from OpenAI Codex.
|
Fresh CI is complete at This update changes only the README, guide, and handoff. The rewrite remains preproduction. The static preflights are not the frozen exact N6 gate, broader format qualification, or authorization to publish or merge this rewrite. AI disclosure: This work was prepared with assistance from OpenAI Codex. |
Align the architecture and roadmap with the implemented recursive nested, shared scalar, statistics, and offset-index paths. Mark target-only modules and API options explicitly and remove the completed documentation handoff item. AI disclosure: This work was prepared with assistance from OpenAI Codex.
An ArrowTypes extension (weak dependency, compat 2, which covers the 2.4 that Arrow.jl's core rewrite carries): with Arrow.jl loaded, `Arrow.write` stores a `Parquet.JSONValue` column as the canonical `arrow.json` extension over its text and a `Parquet.BSONValue` column as plain binary. Arrow.jl reads `arrow.json` back as strings, by its own design. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Polars stores Int128 and UInt128 as FIXED_LEN_BYTE_ARRAY(16) (little-endian)
whose ARROW:schema field is an Int of 128 bits; Parquet.jl read them as
NTuple{16,UInt8}. Such leaves, and leaves the ARROW:schema marks with Arrow.jl's
`JuliaLang.Int128` / `JuliaLang.UInt128` extension, now read as Int128 and
UInt128, flat and nested.
Int128 and UInt128 columns, which were an error, now write the same way as
Arrow.jl's mapping: FIXED_LEN_BYTE_ARRAY(16) with no Parquet annotation (pyarrow
refuses INT(128)) and the extension name in the ARROW:schema field. pyarrow,
Polars, and DuckDB read the column as 16-byte binary. Statistics of these
columns order bytes, not integers, so reads do not prune with them, and the
extension keys stay out of the column metadata. Rewriting a Parquet.Table
keeps nested 128-bit leaves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…does
Typed rows nest one type per level and Julia compiles deep ones slowly. First
reads in a fresh process (Julia 1.12.6) with typed rows to 32 levels: structs
17 deep took 2.5 s to read and 8.6 s more for the first row, lists 17 deep
3.4 s, and both at 32 levels ran past 15 minutes. A column nested more than 8
levels deep (Arrow.jl's threshold) now reads with untyped rows at every level:
structs as StructValue name => value pairs (Arrow's fallback), lists as
ListValue{Any}, and maps as MapValue{Any,Any}. Every depth from 9 to 32 then
reads in about 0.2 s, its first row in 0.01 s. Such rows write back as read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… any order The unit comparison used Dates.toms, which is 0 for both microseconds and nanoseconds, so a dataset of both took the first file's unit and failed with a raw InexactError on the other file's values. Units now compare in nanoseconds, so the finer unit wins whatever the file order, and a value the finer unit cannot hold (a microsecond timestamp in 9999 read as nanoseconds) is an ArgumentError naming the column and the file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A key whose values all parsed as Float64 was a Float64 column, so integers too large for Int64 rounded (12345678901234567890 and ...891 read as one value), and `+7` and `1.10` lost their text. A key is now Int64, Float64, Date, or Bool only when printing each parsed value gives back its text exactly; otherwise it stays a string column, as pyarrow and DuckDB read it. Keys Parquet.write writes still read typed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DuckDB's V2 writer computes INT32 deltas without 32-bit wrapping, so random
INTEGER values or large UINTEGER values give 33-bit miniblocks, and Parquet.jl
refused the column ("delta miniblock bit width 33 exceeds the physical width
32"). Miniblock widths up to 64 are now accepted for INT32 as for INT64; values
already add in 64 bits and keep their low 32, as Polars and DuckDB read them.
A small DuckDB 1.5.5 file is a fixture, checked against DuckDB's own reading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bytes parquet-mr before 1.10 (and parquet-cpp before 1.3) compared byte arrays as signed bytes. For decimals stored as FIXED_LEN_BYTE_ARRAY or BYTE_ARRAY that order is wrong (it puts 2.00 below 1.00), but the trust rules took every decimal comparison as signed and so trusted them: a filter on parquet-testing's fixed_length_decimal.parquet (parquet-mr 1.8.2) dropped the row holding 1.00, and Parquet.statistics reported min 2.00. Decimals stored as bytes now fall to the rule Arrow C++ applies to the other orders of those writers (untrusted unless min == max), so such statistics neither prune nor appear in Parquet.statistics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The i16 ordinal that parquet-java writes wraps negative from row group 32,769, and a negative ordinal made the file unreadable. Nothing reads the ordinal, so it is no longer checked, as pyarrow and DuckDB ignore it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… type As parquet-testing asks for int32_with_uuid_logical_type.parquet, an annotation such as UUID on INT32 is ignored: the column reads as its physical type, its statistics are not used or reported, and rewriting the table drops the annotation, so pyarrow reads the rewrite. Wrong parameters (a DECIMAL precision above its storage, an unknown time unit) stay format errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olds them As pyarrow and DuckDB read them (and Arrow.jl, A7), a DECIMAL(10, 2) INT64 value of 92233720368547758.07 now reads as stored instead of failing the column. A value the column's storage integer cannot hold, such as 10^12 in a DECIMAL(9, 2) INT64 column (Int32 storage), throws UnsupportedFeatureError naming the column and how to read the rest. PLAIN pages copied in bulk need no check: they copy only when the storage integer is as wide as the stored one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ners
Parquet.MapValue{K,V} is now an AbstractDict{K,V}, as in Arrow.jl: it iterates
`key => value` entries in stored order, duplicate keys included, which `length`
counts, and `row[key]`, `get`, and `haskey` find the last entry with that key,
the one `Dict(row)` keeps. `keys` and `values` are vectors in stored order;
`collect(row)` gives the entries and `copy(row)` a Dict. Rows no longer index
by position (Arrow.jl 2.x returned Dicts, and its users index map rows by key).
Writes keep the map-row fast path, and untyped map rows still write their
entries in order with duplicates.
Inside containers and under `:compact` (DataFrame cells), ListValue and
MapValue rows print as `[1, missing]` and `Dict("a"=>1)`, without element types.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olds them As pyarrow does, a DECIMAL coefficient with more digits than the precision now writes as stored when the column's INT32, INT64, or FIXED_LEN_BYTE_ARRAY storage holds it, so a file read and written again (compaction) keeps such values. A value the storage cannot hold is still an ArgumentError naming the column. The interop job checks that pyarrow, Polars, and DuckDB read such a rewrite's values exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- read_parquet and write_parquet, the only exported names, had no docstrings. Add them, list them in the API reference, and test that every exported and public name has a docstring. - README: the quick start's last line read a directory it never wrote; write a dataset first. - Type mapping: map columns read back as Parquet.MapValue rows (read-only AbstractDicts), not vectors of pairs. - Migration guide: Tables.Scan takes keywords (offset=, limit=). - Parquet.File: DataAPI.nrow with a filter is an ArgumentError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the unused flat-column selector path and duplicate write leaf wrappers. Writer options now resolve against the schema nodes already validated by the nested planner.
Reuse the existing dynamic element-type boundary for runtime table columns. Loading CategoricalArrays no longer invalidates schema-preserving writer and budget-accounting code through abstract eltype dispatch. The existing fresh-process invalidation regression passes without changing its bound.
Replaces the 0.8 reader and writer with a Julia implementation that reads modern Parquet files, preserves supported nested and logical values, and works through Tables.jl. The package UUID and Git history are retained.
Public behavior
Parquet.Table(source)reads a file or dataset eagerly;Parquet.read(source, sink)passes it to a Tables.jl sink.Parquet.Fileiterates row groups and supportsdoblocks for cleanup.Parquet.Datasetdiscovers files, unifies their schemas, and supports Hive partition columns.Parquet.writeaccepts Tables.jl sources.Parquet.Writerstreams encoded chunks in file order; regular-file output replaces the destination only after completing the file. Partitioned output has file-level atomicity, not a dataset transaction.Architecture and review changes
The implementation separates byte sources and ownership, generated format metadata, schema validation, typed column kernels, read planning, and streaming output.
CONTRIBUTING.mddocuments these boundaries and their contracts.The review recovered the final unpushed documentation commit and fixed concrete correctness and resource issues:
Dataset discovery deliberately reads every file's footer to establish a stable unified schema, including files whose data a partition filter later skips. Narrow the input paths or prefix to avoid that discovery work.
Validation
Validation is pinned to
a177f289bbec4cbba047b2ff45e5f31930475029.A paired compilation experiment used Julia 1.12.6, two threads, identical dependencies, and two differently named 10,000-row × 1,000-column schemas. On the second schema, constructing a DataFrame took 13.578 s before and 0.0138 s after the table refactor; rewriting to Parquet took 23.053 s and 0.0620 s; writing to Arrow took 36.167 s and 0.0624 s. The old timings were almost entirely compilation; the second schema incurred no measured compilation in these three consumers after the change. Values were checked through both rewrites; the Arrow comparison used the same local Arrow 3.0.0-DEV commit
689cb9cin both environments. These are single fresh-process measurements on a shared host, not steady-state throughput claims. Separately, warmed option-planning allocations for 1,000 writer leaves fell 16.5% (minimum of five samples).Limits
The resource budget bounds package-managed read materialization, not total process memory. Parallel writes can retain encoded chunks from several row groups and keep page indexes and metadata until close. Encryption, LZO, and external column-chunk files are unsupported; the limits guide describes reduced support for INT96, VARIANT, and geospatial storage. Live cloud services and crash/power-loss durability have not been validated by this review.
Co-authored by Codex