GH-531: Add parquet flatbuf schema by alkis · Pull Request #544 · apache/parquet-format

alkis · 2025-12-12T07:58:13Z

Rationale for this change

Improve wide table support.

What changes are included in this PR?

Add parquet flatbuf schema.

Do these changes have PoC implementations?

apache/arrow#48431

emkornfield · 2026-01-09T18:19:39Z

src/main/flatbuf/parquet3.fbs

@@ -0,0 +1,224 @@
+namespace parquet.format3;


lets just name this parquet.format for now?

emkornfield · 2026-01-09T18:20:45Z

src/main/flatbuf/parquet3.fbs

+// 1. Statistics are stored in integral types if their size is fixed, otherwise prefix + suffix
+// 2. ColumnMetaData.encoding_stats are removed, they are replaced with
+//    ColumnMetaData.is_fully_dict_encoded.
+// 3. RowGroups are limited to 2GB in size, so we can use int for sizes.


I think this and the item below are out of date (we are using long now) and can keep things absolute?

emkornfield · 2026-01-09T18:21:08Z

src/main/flatbuf/parquet3.fbs

@@ -0,0 +1,224 @@
+namespace parquet.format3;
+
+// Optimization notes


Can we expand this comment to be explicit about the relationship between this FBS and parquet.thrift.

emkornfield · 2026-01-09T18:22:00Z

src/main/flatbuf/parquet3.fbs

+// Note: Match the thrift enum values so that we can cast between them.
+enum Encoding : byte {
+  PLAIN = 0,
+  // GROUP_VAR_INT = 1,


Call out commented out entries as deprecated to make it clear why they are commented out?

emkornfield · 2026-01-09T18:22:09Z

src/main/flatbuf/parquet3.fbs

+  GZIP = 2,
+  LZO = 3,
+  BROTLI = 4,
+  // LZ4 = 5,


same comment on deprecation.

emkornfield · 2026-01-09T18:22:49Z

src/main/flatbuf/parquet3.fbs

+  scale: int;
+}
+enum TimeUnit : byte {
+  MS = 0,


Can we please make these match parquet.thrift for names (Millisecond, Microsecond, Nanosecond)?

emkornfield · 2026-01-09T18:23:30Z

src/main/flatbuf/parquet3.fbs

+// Logical types.
+///////////////////////////////////////////////////////////////////////////////////////////////////
+
+table Empty {}


I think we want detailed docs (same level as parquet.thrift if we intend this to be the new footer)?

emkornfield · 2026-01-09T18:24:00Z

src/main/flatbuf/parquet3.fbs

+///////////////////////////////////////////////////////////////////////////////////////////////////
+
+table Empty {}
+table DecimalOpts {


Suggested change

table DecimalOpts {

table DecimalOptions {

Should be spell out type names to make it easer on reader?

emkornfield · 2026-01-09T18:24:48Z

src/main/flatbuf/parquet3.fbs

+  // - BYTE_ARRAY:
+  //   prefix: the longest common prefix of min/max
+  //   lo8+hi8 zero padded 16 bytes (big-endian) of the suffix
+  //   len: the length for the suffix of the value after removing the prefix. If > 16 then the


Suggested change

// len: the length for the suffix of the value after removing the prefix. If > 16 then the

// min_len/max_len: the length for the suffix of the value after removing the prefix. If > 16 then the

emkornfield · 2026-01-09T18:26:00Z

src/main/flatbuf/parquet3.fbs

+  //   prefix: the longest common prefix of min/max
+  //   lo8+hi8 zero padded 16 bytes (big-endian) of the suffix
+  //   len: the length for the suffix of the value after removing the prefix. If > 16 then the
+  //        value is inexact


Suggested change

// value is inexact

// value is inexact (it is exact otherwise).

emkornfield · 2026-01-09T18:28:28Z

src/main/flatbuf/parquet3.fbs

+  // - BOOLEAN: none
+  // - INT32/FLOAT: lo4 (little-endian)
+  // - INT64/DOUBLE: lo8 (little-endian)
+  // - INT96: lo4+lo8 (little-endian)


for composite values, I think this is complicated enough that providing concrete examples would be belpful for implementors?

emkornfield · 2026-01-09T18:28:53Z

src/main/flatbuf/parquet3.fbs

+  DATA_PAGE_V2 = 3,
+}
+
+table KV {


nit

Suggested change

table KV {

table KeyValue {

Lets keep name consistent if possible?

emkornfield · 2026-01-09T18:29:51Z

src/main/flatbuf/parquet3.fbs

+  codec: CompressionCodec;
+  num_values: long = null;  // only present if not equal to rg.num_rows
+  total_uncompressed_size: long;
+  total_compressed_size: long;


It would be nice to keep total unencoded size here which I think is generally useful? But I suppose it can be added after?

emkornfield · 2026-01-09T18:30:26Z

src/main/flatbuf/parquet3.fbs

+  dictionary_page_offset: long = null;
+  statistics: Statistics;
+  is_fully_dict_encoded: bool;
+  bloom_filter_offset: long = null;


Should we this be made a struct/value to make the bloom filter info more self contained?

emkornfield · 2026-01-09T18:30:54Z

src/main/flatbuf/parquet3.fbs

+  row_groups: [RowGroup];
+  kv: [KV];
+  created_by: string;
+  // column_orders: [ColumnOrder];  // moved to SchemaElement


remove this row for now?

emkornfield

I think we also need to add an apache header here, and CI to make sure this compiles?

Jiayi-Wang-db · 2026-03-03T12:16:05Z

Hi @rok and @emkornfield , could you help to have another look of this pr?

rok · 2026-03-10T22:40:04Z

src/main/flatbuf/parquet3.fbs

+// It can currently be attached as a footer extension, and may fully replace the
+// Thrift footer in the future.


Maybe we should explicitly state thrift footer is still required as of now?

rok · 2026-03-10T23:39:26Z

src/main/flatbuf/parquet3.fbs

+
+  /** Optional Bloom filter information for this column chunk */
+  bloom_filter: BloomFilterInfo;
+}


Currently this proposal omits optional GeospatialStatistics which with bounding boxes (8 doubles) makes for nice pruning metadata and would likely be worth keeping.

parquet-format/src/main/thrift/parquet.thrift

Line 944 in 38818fa

17: optional GeospatialStatistics geospatial_statistics;

rok · 2026-03-11T00:05:18Z

src/main/flatbuf/parquet3.fbs

+  min_lo4: uint;
+  min_lo8: ulong;
+  min_hi8: ulong;
+  min_len: byte = null;


Suggested change

min_len: byte = null;

min_len: int = null;

Original suffix lenght could exceed int8 range of byte type.

rok · 2026-03-11T00:05:34Z

src/main/flatbuf/parquet3.fbs

+  max_lo4: uint;
+  max_lo8: ulong;
+  max_hi8: ulong;
+  max_len: byte = null;


As above:

Suggested change

max_len: byte = null;

max_len: int = null;

rok · 2026-03-11T00:09:35Z

src/main/flatbuf/parquet3.fbs

+}
+
+table Statistics {
+  null_count: int = null;


Suggested change

null_count: int = null;

null_count: long = null;

To match row group size range.

rok · 2026-03-11T00:16:39Z

src/main/flatbuf/parquet3.fbs

+
+union ColumnCryptoMetadata {
+  EncryptionWithFooterKey:Empty,
+  EncryptionWithColumnKey:Empty,
+}


Used by readers to recover the column key from KMS services.

Suggested change

union ColumnCryptoMetadata {

EncryptionWithFooterKey:Empty,

EncryptionWithColumnKey:Empty,

}

table EncryptionWithColumnKey {

/** Column path in schema **/

path_in_schema: [string];

/** Retrieval metadata of column encryption key **/

key_metadata: [byte];

}

union ColumnCryptoMetadata {

EncryptionWithFooterKey:Empty,

EncryptionWithColumnKey:EncryptionWithColumnKey,

}

rok · 2026-03-11T00:22:44Z

src/main/flatbuf/parquet3.fbs

+
+  /** repetition of the field. The root of the schema does not have a repetition_type.
+   * All other nodes must have one */
+  repetition_type: FieldRepetitionType;


To allow for root to not have repetition type. In thrift we have optional:

parquet-format/src/main/thrift/parquet.thrift

Line 518 in 38818fa

3: optional FieldRepetitionType repetition_type;

Suggested change

repetition_type: FieldRepetitionType;

repetition_type: FieldRepetitionType = null;

rok · 2026-03-11T00:29:35Z

src/main/flatbuf/parquet3.fbs

+  // - INT32/FLOAT: min_lo4/max_lo4 (little-endian, 4 bytes)
+  // - INT64/DOUBLE: min_lo8/max_lo8 (little-endian, 8 bytes)
+  // - INT96: lo4 contains the low 4 bytes, lo8 contains the high 8 bytes (little-endian, 12 bytes total)
+  // - FIXED_LEN_BYTE_ARRAY:


Perhaps note it's the same as for BYTE_ARRAY?

Suggested change

// - FIXED_LEN_BYTE_ARRAY:

// - FIXED_LEN_BYTE_ARRAY: Encoded the same way as BYTE_ARRAY below

rok · 2026-03-11T00:39:48Z

src/main/flatbuf/parquet3.fbs

+  max_lo8: ulong;
+  max_hi8: ulong;
+  max_len: byte = null;
+  prefix: string;


Shouldn't this be binary?

rok · 2026-03-11T00:46:59Z

src/main/flatbuf/parquet3.fbs

+table DecimalOptions {
+  precision: int;
+  scale: int;
+}


In thrift these are in reverse order. Any reason to change that here?

parquet-format/src/main/thrift/parquet.thrift

Lines 343 to 346 in 38818fa

struct DecimalType {

1: required i32 scale

2: required i32 precision

}

rok · 2026-03-11T00:49:27Z

src/main/flatbuf/parquet3.fbs

+// 1. Statistics use fixed-width integral types when possible; otherwise they are
+//    encoded as prefix + suffix.


SizeStatistics and Statistics.distinct_count are removed.

rok · 2026-03-11T17:05:33Z

@Jiayi-Wang-db Feel free to resolve comments you feel were addressed to make this more readable?

Add parquet flatbuf schema

f951a6d

This was referenced Jan 7, 2026

Parquet metadata as flatbuffers apache/arrow-rs#9041

Open

parquet: use flatbuffers to store metadata (WIP) apache/arrow-rs#9042

Draft

emkornfield reviewed Jan 9, 2026

View reviewed changes

src/main/flatbuf/parquet3.fbs Outdated

GZIP = 2,

LZO = 3,

BROTLI = 4,

// LZ4 = 5,

Copy link

Contributor

emkornfield Jan 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment on deprecation.

emkornfield reviewed Jan 9, 2026

View reviewed changes

emkornfield requested changes Jan 9, 2026

View reviewed changes

address comments

a77d277

rok suggested changes Mar 11, 2026

View reviewed changes

address comments - Add GeospatialStatistics, type change in Statistics

a2db68c

		@@ -0,0 +1,224 @@
		namespace parquet.format3;

		// Optimization notes

	// len: the length for the suffix of the value after removing the prefix. If > 16 then the
	// min_len/max_len: the length for the suffix of the value after removing the prefix. If > 16 then the

	// value is inexact
	// value is inexact (it is exact otherwise).

		// It can currently be attached as a footer extension, and may fully replace the
		// Thrift footer in the future.

-union ColumnCryptoMetadata {
-  EncryptionWithFooterKey:Empty,
-  EncryptionWithColumnKey:Empty,
-}
+table EncryptionWithColumnKey {
+  /** Column path in schema **/
+  path_in_schema: [string];
+  /** Retrieval metadata of column encryption key **/
+  key_metadata: [byte];
+}
+union ColumnCryptoMetadata {
+  EncryptionWithFooterKey:Empty,
+  EncryptionWithColumnKey:EncryptionWithColumnKey,
+}

	repetition_type: FieldRepetitionType;
	repetition_type: FieldRepetitionType = null;

	// - FIXED_LEN_BYTE_ARRAY:
	// - FIXED_LEN_BYTE_ARRAY: Encoded the same way as BYTE_ARRAY below

	struct DecimalType {
	1: required i32 scale
	2: required i32 precision
	}

		// 1. Statistics use fixed-width integral types when possible; otherwise they are
		// encoded as prefix + suffix.

Conversation

alkis commented Dec 12, 2025

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield left a comment

Choose a reason for hiding this comment

Uh oh!

Jiayi-Wang-db commented Mar 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants