Skip to content

bug: Support read_parquet glob file paths#34

Merged
jiayuasu merged 3 commits intoapache:mainfrom
petern48:fix_local_glob
Sep 6, 2025
Merged

bug: Support read_parquet glob file paths#34
jiayuasu merged 3 commits intoapache:mainfrom
petern48:fix_local_glob

Conversation

@petern48
Copy link
Contributor

@petern48 petern48 commented Sep 5, 2025

This one was pretty annoying to debug. Here's what was going on. We were calling .to_urls() in twice: once in geoparquet_listing_table and once in here in context.rs's read_parquet:

pub async fn read_parquet<P: DataFilePaths>(
&self,
table_paths: P,
options: GeoParquetReadOptions<'_>,
) -> Result<DataFrame> {
let urls = table_paths.to_urls()?;
let provider =

The second call was causing the ListingTableUrl's glob attribute to being removed. I fixed this my removing one of the calls.

Note: globbing with the * operations still does NOT work on AWS s3 paths. That would need an additional fix / feature. It looks like Datafusion doesn't support globbing s3 paths yet either: It looks like DataFusion doesn't support globbing s3 paths yet either: apache/datafusion#16303

@petern48 petern48 marked this pull request as ready for review September 5, 2025 22:48
@petern48 petern48 requested a review from paleolimbot September 5, 2025 22:48
Copy link
Member

@paleolimbot paleolimbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me that I owe you cookies or chocolate cake someday for debugging this one.

Thank you!

@jiayuasu jiayuasu merged commit 1112f7a into apache:main Sep 6, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants