fix(spark): add default service account fallback for Spark Connect by amh1k · Pull Request #672 · kubeflow/sdk

amh1k · 2026-07-27T08:51:57Z

What this PR does / why we need it:

This change introduces a fallback to the spark-operator-spark default service account when driver.service_account is not explicitly specified in the Spark Connect API.

This simplifies basic SDK use cases on the default operator installation, where manual specification of a service account on the API becomes cumbersome for end users.

Note: This is a temporary workaround and is meant to be removed once #617 lands.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Fixes #668

Checklist:

Ran make verify locally to ensure code quality, formatting compliance, and no linting errors.
Ran make test-python locally to ensure unit tests pass.
Docs included if any changes are user facing (Not applicable for this internal fallback logic).

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

google-oss-prow · 2026-07-27T08:52:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

github-actions · 2026-07-27T08:52:10Z

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

tariq-hasan · 2026-07-28T12:16:14Z

/ok-to-test
/retest

tariq-hasan

Hi @amh1k! Thanks for raising the PR. I have added a few comments.

Please also sign the commits to pass the DCO check.

tariq-hasan · 2026-07-28T12:39:25Z

        memory=memory,
        template=template,
    )



Please add a pytest parametrized test in utils_test.py to cover all test cases for get_spark_connect_driver_spec.

Working on adding the test

have added the test

Goku2099 · 2026-07-28T17:49:05Z

-                containers=[],
-                service_account_name=driver.service_account,
-            )
+    # Fallback to the the spark-operator chart's default service account if not specified.


Typo in the new comment - "the the spark-operator chart's..." should just be "the spark-operator chart's..."

Goku2099 · 2026-07-28T17:50:09Z

+                "expected_cores": 2,
+                "expected_memory": "4g",
+                "expected_service_account": "custom-spark-sa",
+            },


Would you mind adding one more case where service_account is set but resources is left default?

Goku2099 · 2026-07-28T18:22:25Z

@amh1k Can you please sign your commit to get the DCO check to pass?

This change introduces a fallback to the `spark-operator-spark` default service account when `driver.service_account` is not explicitly specified in the Spark Connect API. Why this is needed: This simplifies simple SDK use cases on the default operator installation, where manual specification of a service account on the API becomes cumbersome. Note: This is a temporary workaround and includes a TODO comment to ensure it is removed once issue kubeflow#617 is fully resolved. Signed-off-by: amh1k <abdulmoizx97@gmail.com>

Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Abdul Moiz Hussain <abdulmoizx97@gmail.com>

Signed-off-by: amh1k <abdulmoizx97@gmail.com>

Copilot AI review requested due to automatic review settings July 27, 2026 08:51

Copilot AI reviewed Jul 27, 2026

google-oss-prow Bot requested review from Electronic-Waste, astefanutti and szaher July 27, 2026 08:52

google-oss-prow Bot added the size/S label Jul 27, 2026

google-oss-prow Bot added the ok-to-test label Jul 28, 2026

tariq-hasan reviewed Jul 28, 2026

View reviewed changes

amh1k force-pushed the fix/spark-connect-default-sa-fallback branch from 15bd5ef to 8388173 Compare July 28, 2026 15:29

google-oss-prow Bot added size/M and removed size/S labels Jul 28, 2026

Goku2099 reviewed Jul 28, 2026

View reviewed changes

amh1k and others added 4 commits July 28, 2026 23:33

Update kubeflow/spark/backends/kubernetes/utils.py

2159288

Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Abdul Moiz Hussain <abdulmoizx97@gmail.com>

test(spark): cover Spark Connect driver spec cases

937b6d5

Signed-off-by: amh1k <abdulmoizx97@gmail.com>

test(spark): cover driver service account defaults

7ea0e33

Signed-off-by: amh1k <abdulmoizx97@gmail.com>

amh1k force-pushed the fix/spark-connect-default-sa-fallback branch from af52c17 to 7ea0e33 Compare July 28, 2026 18:35

Uh oh!

Conversation

amh1k commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow Bot commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

tariq-hasan commented Jul 28, 2026

Uh oh!

tariq-hasan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tariq-hasan Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

amh1k Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

amh1k Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Goku2099 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

amh1k Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Goku2099 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

amh1k Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Goku2099 commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants