[AURON#1404] Support for Spark 4.0.1 Compatibility in Auron. #1923
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1404.
Rationale for this change
[AURON#1404] Support for Spark 4.0.1 Compatibility in Auron.
What changes are included in this PR?
To support Spark 4, Auron needs to be adapted accordingly. Currently,
Celebornalready supports Spark 4.0, andIceberghas also supported Spark 4.0 for some time. The Iceberg community has already voted to deprecate support for Spark 3.4, and it will be removed soon.For this PR, I have made the following changes:
Three changes encountered during compilation:
NativeShuffleExchangeExec#ShuffleWriteProcessor: Due to SPARK-44605 restructuring the write method in the API, I refactored the partition and rdd handling here to retrieve them from dependencies for compatibility with other interfaces. In the future, we should switch to the new interface and make further changes to nativeRssShuffleWrite / nativeShuffleWrite.NativeBroadcastExchangeBase#getBroadcastTimeout: In Spark 4.0, getBroadcastTimeout needs to be fetched from getActiveSession.NativeBroadcastExchangeBase#getRelationFuture: In Spark 4.0, the type of SparkSession has changed to org.apache.spark.sql.classic.SparkSession, so I made the necessary adjustments to the way it is accessed.Are there any user-facing changes?
No.
How was this patch tested?
CI.