fix: Percent Encoding of paths for hive style partitioning #19651
+68
−8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
martin-g
reviewed
Jan 6, 2026
| file_path: &'a Path, | ||
| table_partition_cols: I, | ||
| ) -> Option<Vec<&'a str>> | ||
| ) -> Option<Vec<String>> |
Member
There was a problem hiding this comment.
nit: Not sure whether it is worth it to return a Vec<Cow<'a, str>> here ?! E.g. if the partition contains % then decode it and return Cow::Owned(decoded.into_owned()), otherwise `Cow::Borrowed(val).
Member
There was a problem hiding this comment.
Actually this is what https://docs.rs/percent-encoding/latest/percent_encoding/struct.PercentDecode.html#method.decode_utf8 returns. It might be a good idea.
| match part.split_once('=') { | ||
| Some((name, val)) if name == expected_partition => part_values.push(val), | ||
| Some((name, val)) if name == expected_partition => { | ||
| let decoded = percent_decode_str(val).decode_utf8().ok()?; |
Member
There was a problem hiding this comment.
Should invalid values (like %FF in one of the tests below) return None or val ?!
I think it should behave like %XX - i.e. return val
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Currently, when DataFusion parses Hive-style partitioned paths (e.g.,
s3://bucket/table/city=San%20Francisco/), it extracts the partition value literally asSan%20Francisco.Standard practice in tools like Apache Spark and Apache Hive is to URL-encode special characters in partition values when writing to object stores. This PR ensures DataFusion correctly decodes these values (to
San Francisco) during the listing process, preventing data mismatches and ensuring consistent behavior with other engines.What changes are included in this PR?
parse_partitions_for_pathindatafusion/catalog-listing/src/helpers.rsto percent-decode partition values.parse_partitions_for_pathfromOption<Vec<&str>>toOption<Vec<String>>because decoded strings require new allocations.datafusion-catalog-listingto accommodate the ownedStringreturn type.percent-encodingcrate todatafusion-catalog-listing.Are these changes tested?
Yes, new unit tests were added to
datafusion/catalog-listing/src/helpers.rscovering:%2Ffor/).%20.percent-encodingcrate defaults).Are there any user-facing changes?
Yes:
parse_partitions_for_pathnow returnsVec<String>instead ofVec<&str>.