feat: [KEP-107] Support options in submit_job() for SparkApplication jobs

[Goku2099]

Summary

This PR adds support for configuring SparkApplication jobs through the options parameter in SparkClient.submit_job().

The following SparkApplication configurations are now supported through options:

• Labels
• Annotations
• NodeSelector
• Toleration
• DriverOption
• ExecutorOption

Additional Changes

• Added unit tests covering the new functionality.
• Added an end-to-end example demonstrating SparkApplication configuration through options.

ref - #520

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign astefanutti for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Goku2099]

cc @tariq-hasan @Shekharrajak @andreyvelich

[tariq-hasan]

/ok-to-test
/retest

[tariq-hasan]

Hi @Goku2099! Thanks for raising the PR! I have added a few comments.

[andreyvelich]

Thanks a lot for working on this, @Goku2099! 🙏

This review was performed with the help of AI tools. Please see the inline comments.

[andreyvelich]

Spark tests shouldn't import trainer test helpers — kubeflow/spark/test/common.py has an identical TestCase (including expected_error) and is what every other spark test uses.

Suggested change

	from kubeflow.trainer.test.common import FAILED, SUCCESS, TestCase
	from kubeflow.spark.test.common import FAILED, SUCCESS, TestCase

[andreyvelich]

Can we move them to kubeflow.common ?

[Goku2099]

@andreyvelich Good point. Since this involves reorganizing shared test utilities across multiple components, I'd prefer to keep it separate from this feature PR. I can open a follow-up issue to move the common test helpers into a shared module and update the imports accordingly.

[Goku2099]

@andreyvelich I've created a tracking issue for consolidating the shared test helpers into kubeflow.common: #700 . Since this involves shared infrastructure across multiple SDK components, I'd prefer to keep it separate from this feature PR.

[Goku2099]

@andreyvelich - I removed the pod template override support and simplified the implementation accordingly. I also updated the batch job options example to keep it focused on demonstrating the options API: it now uses a unique job name for idempotent execution and verifies that the configured Name, Labels, Annotations, NodeSelector, and Toleration are applied to the created SparkApplication.

[Goku2099]

Hi @andreyvelich , I've addressed all of the review comments, Could you please take another look when you have a chance? Thanks!