Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added Shims for adding Databricks 14.3 Support [databricks] #11635

Merged
merged 23 commits into from
Nov 13, 2024
Merged

Added Shims for adding Databricks 14.3 Support [databricks] #11635

merged 23 commits into from
Nov 13, 2024
+6,976 −177

Conversation

razajafri
Copy link
Collaborator

This PR adds Shims for Databricks 14.3

@razajafri
Added 350db shims
8f7be42 
Signed-off-by: Raza Jafri <[email protected]>
@sameerz sameerz added the feature request New feature or request label Oct 19, 2024
@razajafri
Copy link
Collaborator Author

build

@mythrocks mythrocks mentioned this pull request Oct 21, 2024
Closed
@razajafri
Copy link
Collaborator Author

@gerashegalov @mythrocks PTAL

@gerashegalov gerashegalov self-requested a review October 30, 2024 00:26
gerashegalov
gerashegalov requested changes Oct 30, 2024
View reviewed changes
Copy link
Collaborator

@gerashegalov gerashegalov left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please upmerge

I cannot build this PR branch. It seems to be missing the release350db profile etc.

datagen/src/main/spark320/scala/org/apache/spark/sql/tests/datagen/DataGenExprShims.scala Outdated Show resolved Hide resolved
...ugin/src/main/spark350db/scala/com/nvidia/spark/rapids/spark350db/RapidsShuffleManager.scala Outdated
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file should be dropped after upmerging to branch-24.12

...rc/main/scala/com/databricks/sql/transaction/tahoe/rapids/GpuOptimisticTransactionBase.scala Outdated Show resolved Hide resolved
@gerashegalov
Copy link
Collaborator

Please update Scala 2.13 poms

@razajafri
Copy link
Collaborator Author

@gerashegalov I have addressed your concerns. PTAL

gerashegalov
gerashegalov reviewed Nov 2, 2024
View reviewed changes
Copy link
Collaborator

@gerashegalov gerashegalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. at least one copyright to update

sql-plugin/src/main/spark320/scala/com/nvidia/spark/rapids/RapidsShuffleManager.scala Outdated Show resolved Hide resolved
gerashegalov
gerashegalov previously approved these changes Nov 4, 2024
View reviewed changes
@razajafri
Copy link
Collaborator Author

build

@mythrocks
Copy link
Collaborator

It's looking good to go, now that the build is passing as well, no?

@NvTimLiu
Copy link
Collaborator

NvTimLiu commented Nov 7, 2024

@razajafri Can you enable Databricks-14.3 in the pre-merge CI pipeline in this PR as below?

https://github.com/NVIDIA/spark-rapids/blob/branch-24.12/jenkins/Jenkinsfile-blossom.premerge-databricks#L94

values '11.3', '12.2', '13.3', '14.3'

mythrocks
mythrocks previously approved these changes Nov 7, 2024
View reviewed changes
@razajafri
Copy link
Collaborator Author

build

@razajafri
Copy link
Collaborator Author

@razajafri Can you enable Databricks-14.3 in the pre-merge CI pipeline in this PR as below?

https://github.com/NVIDIA/spark-rapids/blob/branch-24.12/jenkins/Jenkinsfile-blossom.premerge-databricks#L94

values '11.3', '12.2', '13.3', '14.3'

I have updated it but this will turn the integration tests on. We don't want the integration tests to run against Databricks 14.3 because we aren't ready.

@razajafri
Copy link
Collaborator Author

build

1 similar comment
@razajafri
Copy link
Collaborator Author

build

@razajafri
Copy link
Collaborator Author

build

AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE

@razajafri
Copy link
Collaborator Author

build

1 similar comment
@pxLi
Copy link
Collaborator

pxLi commented Nov 11, 2024

build

@pxLi
Copy link
Collaborator

pxLi commented Nov 11, 2024 •
edited
Loading

build

AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE

This change requires 8 clusters (4 shims x 2 parallelism) for the CI. We're considering using some escalation strategy, such as switching to different instance types or cloud providers or temporarily disabling parallel runs for Databricks runtimes.

also cc @NvTimLiu

BTW my last trigger works fine

@razajafri razajafri merged commit 57b8caa into NVIDIA:branch-24.12 Nov 13, 2024
49 checks passed
@razajafri razajafri deleted the SP-10661-db-14.3-2 branch November 13, 2024 18:34
@razajafri razajafri restored the SP-10661-db-14.3-2 branch November 14, 2024 02:15
@razajafri razajafri deleted the SP-10661-db-14.3-2 branch November 14, 2024 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jlowe jlowe jlowe left review comments

@gerashegalov gerashegalov gerashegalov approved these changes

@revans2 revans2 Awaiting requested review from revans2 revans2 is a code owner

@tgravescs tgravescs Awaiting requested review from tgravescs tgravescs is a code owner

@GaryShen2008 GaryShen2008 Awaiting requested review from GaryShen2008 GaryShen2008 is a code owner

@NvTimLiu NvTimLiu Awaiting requested review from NvTimLiu NvTimLiu is a code owner

@mythrocks mythrocks Awaiting requested review from mythrocks

Assignees

@razajafri razajafri

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@razajafri @gerashegalov @mythrocks @NvTimLiu @pxLi @jlowe @sameerz