What you would like to be added?
Right now SparkClient can tell you the status of a job (get_job, wait_for_job_status) but nothing about what's actually going on inside it while it runs, no executor count, no task progress, nothing. If you want that today you're stuck going through kubectl or opening the Spark UI yourself.
I'd like to add a get_job_metrics(name) method that pulls basic runtime metrics straight from Spark's own REST API on the driver:
GET /api/v1/applications/<app-id>/executors for executor count, active/completed tasks, memory usedGET /api/v1/applications/<app-id>/stages for stage progress
Went through the actual Spark Operator source (web_ui.go, pkg/util/sparkapplication.go) before settling on an approach, a couple of things worth calling out:
The operator creates a dedicated Kubernetes Service for the driver UI on every SparkApplication, default port name spark-driver-ui-port, default port 4040 (driven by spark.ui.port, but can be overridden via SparkUIOptions on the SparkApplication spec). So the port should be read off the live Service object, not assumed.
The service name itself isn't safe to construct by hand either. The operator builds it as <app-name>-ui-svc only when that's 63 characters or under (Kubernetes' DNS label limit), otherwise it truncates the app name and appends an 8 character hash. What is reliable is the label the operator puts on every UI service it creates, sparkoperator.k8s.io/app-name: <app-name>. So the plan is to look the service up by that label selector instead of guessing the name.
Also, this UI service isn't guaranteed to exist at all. It's gated behind an operator-level flag (--enable-ui-service, defaults to true) that whoever deployed the Spark Operator in a given cluster could turn off. So get_job_metrics needs to treat "no UI service for this job" as a normal, expected case, not just something that happens after the job finishes and gets cleaned up.
To reach the service:
- In-cluster: hit it directly through the Service DNS name.
- Outside cluster (local dev, tests): port-forward to it. Same port-forward mechanism the Kubernetes backend already uses for Spark Connect (
get_connect_url in backend.py, around lines 435-511), just pointed at this service instead.
Rough shape of what I'm thinking:
@dataclass
class SparkJobMetrics:
num_executors: int | None = None
active_tasks: int | None = None
completed_tasks: int | None = None
failed_tasks: int | None = None
active_stages: int | None = None
completed_stages: int | None = Noneget_job_metrics should return a mostly-empty SparkJobMetrics rather than raising whenever the UI service can't be found, whether that's because it was disabled at the operator level or because it's already been cleaned up after the job finished.
Why is this needed?
This is part of the observability gap called out in #655, batch jobs work fine but there's basically no visibility into them once they're running. Metrics collection is one of the pieces on that epic that nobody's picked up yet.
Love this feature?
Give it a 👍 We prioritize the features with most 👍
What you would like to be added?
Right now
SparkClientcan tell you the status of a job (get_job,wait_for_job_status) but nothing about what's actually going on inside it while it runs, no executor count, no task progress, nothing. If you want that today you're stuck going through kubectl or opening the Spark UI yourself.I'd like to add a
get_job_metrics(name)method that pulls basic runtime metrics straight from Spark's own REST API on the driver:GET /api/v1/applications/<app-id>/executorsfor executor count, active/completed tasks, memory usedGET /api/v1/applications/<app-id>/stagesfor stage progressWent through the actual Spark Operator source (
web_ui.go,pkg/util/sparkapplication.go) before settling on an approach, a couple of things worth calling out:The operator creates a dedicated Kubernetes Service for the driver UI on every SparkApplication, default port name
spark-driver-ui-port, default port4040(driven byspark.ui.port, but can be overridden viaSparkUIOptionson the SparkApplication spec). So the port should be read off the live Service object, not assumed.The service name itself isn't safe to construct by hand either. The operator builds it as
<app-name>-ui-svconly when that's 63 characters or under (Kubernetes' DNS label limit), otherwise it truncates the app name and appends an 8 character hash. What is reliable is the label the operator puts on every UI service it creates,sparkoperator.k8s.io/app-name: <app-name>. So the plan is to look the service up by that label selector instead of guessing the name.Also, this UI service isn't guaranteed to exist at all. It's gated behind an operator-level flag (
--enable-ui-service, defaults to true) that whoever deployed the Spark Operator in a given cluster could turn off. Soget_job_metricsneeds to treat "no UI service for this job" as a normal, expected case, not just something that happens after the job finishes and gets cleaned up.To reach the service:
get_connect_urlinbackend.py, around lines 435-511), just pointed at this service instead.Rough shape of what I'm thinking:
get_job_metricsshould return a mostly-emptySparkJobMetricsrather than raising whenever the UI service can't be found, whether that's because it was disabled at the operator level or because it's already been cleaned up after the job finished.Why is this needed?
This is part of the observability gap called out in #655, batch jobs work fine but there's basically no visibility into them once they're running. Metrics collection is one of the pieces on that epic that nobody's picked up yet.
Love this feature?
Give it a 👍 We prioritize the features with most 👍