[Tracking]: Spark Client Batch Jobs, Lifecycle Management, Observability, and Production Readiness

[Goku2099]

Tracking Issue: Spark Client Batch Jobs, Lifecycle Management, Observability, and Production Readiness

Overview

This tracking issue tracks implementation work for Spark batch jobs, lifecycle management, observability, testing, documentation, and production readiness for the Spark Client.

Related:

• KEP-107: Spark Client

Design & Proposal

• SparkClient long-running job submission proposal (extends KEP-107) feat: [KEP-107] Add long-running job submission proposal for SparkClient #524

Implementation

Batch Job Submission

• FileJob-based batch job submission, lifecycle management APIs, and unit tests (submit_job(), list_jobs(), get_job(), wait_for_job_status(), delete_job(), get_job_logs()) feat: [KEP-107] Add Spark batch job submission and lifecycle APIs #521
• FuncJob support (submit_job(job=FuncJob(...))) feat: [KEP-107] add FuncJob support for Python function submission #629
• spark_conf support for SparkApplication jobs KEP-107: Wire spark_conf through submit_job() for batch SparkApplication jobs #646
• options support for SparkApplication jobs feat: [KEP-107] Support options in submit_job() for SparkApplication jobs #685

Spark UI & Job Visibility

• Spark UI integration and job visibility improvements #XXX
• Spark History Server integration #XXX

Observability & Monitoring

• Spark metrics integration #XXX
• Kubeflow-specific monitoring and diagnostics #XXX

Testing & Validation

• End-to-end testing across supported Spark versions (3.5.x, 4.x) #XXX

Documentation & Examples

• SparkClient batch jobs documentation and examples #XXX
• Debugging and troubleshooting guide #XXX

[Goku2099]

cc @tariq-hasan @Shekharrajak @andreyvelich

[andreyvelich]

/area spark
@Goku2099 Please could you also update this KEP to support Spark Batch jobs: https://github.com/kubeflow/sdk/tree/main/docs/proposals/107-spark-client ?

[RobuRishabh]

Filed #646 for spark_conf on batch submit_job().