Problem
KubernetesBackend.train() creates a TrainJob CRD via create_namespaced_custom_object() but does not inject any trace context into the CRD metadata. This means SDK-side OTel spans (e.g. kubeflow.sdk.trainer.train) and any future controller-side spans are completely disconnected — the trace stops at the SDK edge.
Where the gap is
In kubeflow/trainer/backends/kubernetes/backend.py, the train() method builds IoK8sApimachineryPkgApisMetaV1ObjectMeta with name, labels, and annotations but never injects opentelemetry.io/traceparent or opentelemetry.io/tracestate into the annotations:
train_job = models.TrainerV1alpha1TrainJob(
metadata=models.IoK8sApimachineryPkgApisMetaV1ObjectMeta(
name=train_job_name, labels=labels, annotations=annotations
),
spec=trainjob_spec,
)Proposed solution
When OTel is active, inject the current span context as W3C Trace Context annotations before CRD creation:
from opentelemetry.context import get_current
from opentelemetry.propagators import inject
# Before creating the TrainJob
carrier = {}
inject(carrier) # injects traceparent + tracestate
if carrier:
annotations = annotations or {}
for k, v in carrier.items():
annotations[f"opentelemetry.io/{k}"] = vThis is zero-overhead when OTel is not installed (guarded by the same get_tracer() pattern from #401). The controller can later extract these annotations to continue the trace.
This pattern is already used in the Kubernetes ecosystem — e.g. Tekton Pipelines propagates traceparent through CRD annotations.
Why a separate issue
This is a specific cross-boundary propagation design question, distinct from the general SDK instrumentation scoped in #164. The annotation key format (opentelemetry.io/traceparent vs env vars vs labels) and controller-side extraction pattern need explicit agreement before implementation.
Relates to #164, KEP #382, PR #401
Problem
KubernetesBackend.train()creates a TrainJob CRD viacreate_namespaced_custom_object()but does not inject any trace context into the CRD metadata. This means SDK-side OTel spans (e.g.kubeflow.sdk.trainer.train) and any future controller-side spans are completely disconnected — the trace stops at the SDK edge.Where the gap is
In
kubeflow/trainer/backends/kubernetes/backend.py, thetrain()method buildsIoK8sApimachineryPkgApisMetaV1ObjectMetawithname,labels, andannotationsbut never injectsopentelemetry.io/traceparentoropentelemetry.io/tracestateinto the annotations:Proposed solution
When OTel is active, inject the current span context as W3C Trace Context annotations before CRD creation:
This is zero-overhead when OTel is not installed (guarded by the same
get_tracer()pattern from #401). The controller can later extract these annotations to continue the trace.This pattern is already used in the Kubernetes ecosystem — e.g. Tekton Pipelines propagates traceparent through CRD annotations.
Why a separate issue
This is a specific cross-boundary propagation design question, distinct from the general SDK instrumentation scoped in #164. The annotation key format (
opentelemetry.io/traceparentvs env vars vs labels) and controller-side extraction pattern need explicit agreement before implementation.Relates to #164, KEP #382, PR #401