feat(spark): convert spark job example to Jupyter notebook

[Ashvatthaa]

Summary

This PR converts the spark_job.py example into a Jupyter notebook.

Changes

• Replaced examples/spark/spark_job.py with examples/spark/spark_job.ipynb
• Organized the workflow into executable notebook cells
• Added markdown explanations for each step
• Preserved the original example behavior

Motivation

Notebook-based examples provide a more interactive learning experience and align the Spark SDK examples with the Trainer examples, making them easier to use in Kubeflow Notebooks.

Fixes #635

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign astefanutti for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[lntutor]

The cell immediately before SparkSession.builder is marked cell_type: "code", but after the ## Create a Spark Session comment it contains the bare text Create a Spark session that will be used to execute the example.. Jupyter will execute that as Python and raise a SyntaxError, so the notebook cannot be run top-to-bottom.

Please make this a Markdown cell (like the other section introductions), or turn the prose into a Python comment, before merging.

[google-oss-prow]

@lntutor: changing LGTM is restricted to collaborators

Details

In response to this:

The cell immediately before SparkSession.builder is marked cell_type: "code", but after the ## Create a Spark Session comment it contains the bare text Create a Spark session that will be used to execute the example.. Jupyter will execute that as Python and raise a SyntaxError, so the notebook cannot be run top-to-bottom.

Please make this a Markdown cell (like the other section introductions), or turn the prose into a Python comment, before merging.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tariq-hasan]

Hi @Ashvatthaa! Thanks for raising the PR.

We should leave examples/spark/spark_job.py unchanged as it is used as a remote URI on one of the Spark e2e test examples.

We would want all the other example modules sitting within examples/spark to be refactored as Jupyter Notebook examples.

[Ashvatthaa]

@lntutor Thanks for the review! I moved the explanatory text into a Markdown cell before the SparkSession.builder code cell and updated the notebook.

[Ashvatthaa]

@tariq-hasan Thanks for the clarification! I'll keep examples/spark/spark_job.py unchanged since it's used by the Spark e2e tests. For this PR, would you prefer that I convert all of the remaining Python example modules under examples/spark into Jupyter notebooks, or would you prefer splitting that work into separate PRs?

[tariq-hasan]

I think best to keep the example changes consolidated in this PR as some change will also be needed towards updating the test harness (sdk/test/e2e/spark, sdk/hack, etc.) to support these notebooks.

[Ashvatthaa]

@tariq-hasan Thanks for the clarification. I investigated the existing notebook testing infrastructure and noticed the repository already uses Papermill via make test-e2e-notebook for notebook e2e tests. For the Spark examples, would you like the Spark e2e harness to reuse that notebook execution approach, or do you have a different mechanism in mind for executing the converted notebooks?

[andreyvelich]

@tariq-hasan Any objections to simple run E2E scripts w/ Jupyter Notebooks, and remove all Python examples from /test?
I think that might be sufficient to verify functionalities and ensure our Notebooks are working.

We do the same for Trainer:
https://github.com/kubeflow/trainer/blob/master/.github/workflows/test-e2e.yaml#L57-L66

[tariq-hasan]

@andreyvelich That make sense to me - given that the Trainer project follows the same pattern.

At this time the Trainer package tests run via the test-e2e workflow while the Spark package tests run via the test-spark-examples workflow.

I was looking to get your thoughts on whether these should be consolidated into a single workflow or whether the workflows should remain separate - especially in the context of how e2e for other packages (pipelines, feast, etc.) need to be maintained as well.

The difference in the two workflows is primarily in the cluster setup process - Trainer's setup is more involved while Spark uses a different setup. I'm thinking any attempt at possible consolidation - whether it's just cluster setup consolidation or something more involved - can be parked away as part of the common infra setup stream of work and we can keep this PR focused on just the Jupyter Notebook conversion for Spark examples - but it would be good to get your thoughts on this.

[Ashvatthaa]

@tariq-hasan Thanks for the review! I have updated the PR by:

Converting the Spark SDK examples into Jupyter notebooks.
Adding notebook execution support via Papermill.
Adding a reusable test-e2e-notebook Make target.
Adding hack/e2e-run-notebook.sh for notebook execution.
Extending the Spark E2E workflow to execute all notebooks.
Updating the in-cluster runner to support both Python scripts and notebooks.

Please let me know if there are any additional changes you'd like me to make.