update default spark version

[nabuskey]

What this PR does / why we need it:

This updates the default spark image / version used by the SDK to 4.0.4 which has many security fixes.

We have updated the version to 4.0.4 here: kubeflow/spark-operator#3052
So I'd like to keep the SDK in-sync with the spark operator repository.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kramaranya for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[tariq-hasan]

I have two points I wanted to mention.

Firstly, I am thinking that aligning the pyspark-connect package with the image version would be good as well - so both at 4.0.4. For context the package version was 4.0.1 before but a dependabot update triggered the bump to 4.2.0.

Secondly, the latest version of the kubeflow-spark-api models is 2.4.0 @andreyvelich. It would probably be great to push a 2.5.x version of the package and reference the new version in pyproject.toml.