Skip to content

Update parquet to 1.15.2 in PXF#52

Merged
ostinru merged 2 commits intoMDB_STABLEfrom
pxf-parquet-1.15.2
Feb 19, 2026
Merged

Update parquet to 1.15.2 in PXF#52
ostinru merged 2 commits intoMDB_STABLEfrom
pxf-parquet-1.15.2

Conversation

@ostinru
Copy link
Copy Markdown
Contributor

@ostinru ostinru commented Feb 19, 2026

Update parquet to 1.15.2

  • Add support for ZSTD compression

  • Add support for LZ4_RAW codec

  • More optimal memory usage in compression codecs

  • Temporary declare LZO as not supported. It causes following error in both current and in pre-upgrade builds:

    ERROR:  PXF server error : Class com.hadoop.compression.lzo.LzoCodec was not found  (seg1 10.11.0.131:6000 pid=2567556)
    
  • Add tests to cover different types of compression

Dependency tree changes are small:

+--- org.apache.parquet:parquet-column:1.15.1
|    +--- org.apache.parquet:parquet-common:1.15.1
|    |    +--- org.apache.parquet:parquet-format-structures:1.15.1
|    +--- org.apache.parquet:parquet-encoding:1.15.1
|    |    +--- org.apache.parquet:parquet-common:1.15.1 (*)
+--- org.apache.parquet:parquet-hadoop:1.15.1
|    +--- org.apache.parquet:parquet-column:1.15.1 (*)
|    +--- org.apache.parquet:parquet-format-structures:1.15.1 (*)
|    +--- org.apache.parquet:parquet-common:1.15.1 (*)
|    +--- org.xerial.snappy:snappy-java:1.1.10.7
|    +--- io.airlift:aircompressor:2.0.2
|    +--- commons-pool:commons-pool:1.6
|    +--- com.github.luben:zstd-jni:1.5.6-6
+--- org.apache.parquet:parquet-jackson:1.15.1
+--- org.apache.parquet:parquet-generator:1.15.1
+--- org.apache.parquet:parquet-pig:1.15.1
|    +--- org.apache.parquet:parquet-column:1.15.1 (*)
|    +--- org.apache.parquet:parquet-hadoop:1.15.1 (*)
|    +--- org.apache.parquet:parquet-common:1.15.1 (*)
\--- org.apache.parquet:parquet-format:2.10.0

It has its own thrift library shaded. It doesn't depend on protobuf.
parquet-hadoop in fact expects that there is hadoop-client, hadoop-common, hadoop-annotations and hadoop-mapreduce-client-core provided.

Update parquet to 1.12.3 (latest version with hadoop-client-2.x)
* Add support for ZSTD compression
* Temporary declare LZO as not supported. It causes following error in both current and in pre-upgrade builds:
   ```
   ERROR:  PXF server error : Class com.hadoop.compression.lzo.LzoCodec was not found  (seg1 10.11.0.131:6000 pid=2567556)
   ```
* Add tests to cover different types of compression
@ostinru
Copy link
Copy Markdown
Contributor Author

ostinru commented Feb 19, 2026

Passed: 1005 Failed: 92 Skipped: 61

Copy link
Copy Markdown

@leborchuk leborchuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ostinru ostinru marked this pull request as ready for review February 19, 2026 09:15
@ostinru ostinru merged commit 20f568c into MDB_STABLE Feb 19, 2026
40 of 65 checks passed
@ostinru ostinru deleted the pxf-parquet-1.15.2 branch February 19, 2026 09:16
ostinru added a commit that referenced this pull request Feb 24, 2026
Add examples on how to use PXF foreign data wrappers to access ORC files on S3/HDFS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants