feat(runner): optional MLflow tracing parallel to Langfuse (#1263)

<!-- acp:session_id=session-f6f5c9bf-a278-4c57-a92b-f648abd59de1
source=#1263 last_action=2026-04-10T14:29:24Z retry_count=3 -->
## Summary

Adds **optional MLflow GenAI tracing** alongside the existing Langfuse
path. Langfuse remains the **default** when `OBSERVABILITY_BACKENDS` is
unset, so current reports, evals, and feedback (`ambient:langfuse_trace`
/ `traceId`) behave as before unless operators opt in.

**Backward compatibility:** `LANGFUSE_MASK_MESSAGES` / shared masking
uses the **same rules as before this PR** (e.g. redact long strings,
allow-listed keys including `metadata` unchanged). Session
`spec.environmentVariables` **cannot** override operator-injected env
vars that come from **SecretKeyRef** (Langfuse + MLflow), so platform
observability config is not bypassed.

## What changed

### Runner (`ambient-runner`)

- **`OBSERVABILITY_BACKENDS`**: comma list `langfuse`, `mlflow`
(default: `langfuse` only).
- **`MLflowSessionTracer`** (`mlflow_observability.py`): mirrors turn /
tool boundaries with `mlflow.start_span`, usage attributes, session
summary span, error cleanup, and flush.
- **`observability_config.py`** / **`observability_privacy.py`**:
backend selection and shared message masking (`LANGFUSE_MASK_MESSAGES`
still applies to both paths).
- **`ObservabilityManager`**: orchestrates Langfuse + MLflow;
**`mlflow_tracing_active`** property; per-backend try/except so one
backend failing does not disable the other.
- **`pyproject.toml`**: optional extra **`mlflow-observability`**
(`mlflow[kubernetes]>=3.11`, `opentelemetry-exporter-otlp`); included in
**`all`** (Docker `[all]` install).
- **README**: MLflow env vars and OTLP (`OTEL_*`,
`MLFLOW_TRACE_ENABLE_OTLP_DUAL_EXPORT`).
- **Tests**: `test_observability_config.py`,
`test_observability_mlflow_integration.py`;
**`_extract_assistant_text`** tolerates missing `claude_agent_sdk` so
pytest passes without the `claude` extra.

### Operator & manifests

- Copies **`ambient-admin-mlflow-observability-secret`** into session
namespaces when **`MLFLOW_TRACING_ENABLED`** is set on the operator
(same pattern as Langfuse).
- Injects **`MLFLOW_TRACING_ENABLED`**, **`MLFLOW_TRACKING_URI`**,
**`MLFLOW_EXPERIMENT_NAME`**, optional **`MLFLOW_TRACKING_AUTH`** /
**`MLFLOW_WORKSPACE`**, **`OBSERVABILITY_BACKENDS`** via `secretKeyRef`.
- **`replaceOrAppendEnvVars`**: entries already using **`ValueFrom`**
(secret-backed) are not replaced by `spec.environmentVariables`, so user
env cannot override Langfuse or MLflow keys wired from secrets.
- When the MLflow observability secret is in use, runner pods use
**`ServiceAccountName: ambient-session-<session>`** with **token
automount** so `MLFLOW_TRACKING_AUTH=kubernetes-namespaced` can supply
JWT + workspace headers (per MLflow 3.11). Session **`Role`** grants
**`experiments`** (`mlflow.kubeflow.org`) for Kubeflow-style workspace
auth (tune for your CRD group).
- Deletes copied secret on session cleanup (mirrors Langfuse rules).
- **`operator-deployment.yaml`**: optional env from the MLflow
observability secret.
- **Example secret**:
`components/manifests/base/ambient-admin-mlflow-observability-secret.yaml.example`.

### Capabilities API

- **`FrameworkCapabilities.tracing`**: e.g. `langfuse`, `mlflow`, or
`langfuse,mlflow` via **`tracing_capability_label`**.

## How to enable (high level)

1. Create **`ambient-admin-mlflow-observability-secret`** in the
operator namespace (see example YAML).
2. Set operator env **`MLFLOW_TRACING_ENABLED=true`** (and ensure the
secret exists) so the operator copies it and wires runner env.
3. Runner image must include MLflow deps (already true for **`[all]`**
builds).

## Testing

- `cd components/runners/ambient-runner && uv run pytest tests/` (full
suite; includes observability + privacy tests).
- Operator: run **`go test ./...`** with a Go toolchain matching
**`go.mod`** (e.g. 1.24+), including
**`TestReplaceOrAppendEnvVarsPreservesValueFrom`**.

## Commits (squash on merge)

Branch includes the feature commit plus follow-ups, e.g. MLflow k8s auth
/ SA automount / experiment RBAC, `ValueFrom` env precedence,
assistant-text test fix, and masking left aligned with historical
Langfuse behaviour—see **`git log`** on the PR branch for the exact
list.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Optional MLflow observability: per-session MLflow tracing, selectable
backends (langfuse, mlflow), backend selection and privacy utilities,
and conditional runner service-account automount when namespaced auth is
used.

* **Bug Fixes**
* Prevents user env vars from overriding operator-managed observability
vars; preserves platform-managed settings and protects injected names.

* **Documentation**
* Expanded docs for multi-backend observability, MLflow setup, OTLP
export, and masking options.

* **Tests**
* Added tests for backend selection, MLflow integration, and privacy
masking.

* **Chores**
* Example MLflow secret manifest, RBAC for per-session MLflow access,
and packaging extra for MLflow.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Edson Tirelli <etirelli@redhat.com>
Signed-off-by: Ambient Code Bot <bot@ambient-code.local>
Co-authored-by: Ambient Code Bot <bot@ambient-code.local>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: 3 people

components/manifests/base/ambient-admin-mlflow-observability-secret.yaml.example

@@ -0,0 +1,31 @@
+# Example: platform-wide MLflow tracing credentials for the operator to copy into session namespaces.
+# Apply to the operator namespace (same pattern as ambient-admin-langfuse-secret).
+#
+#   kubectl create secret generic ambient-admin-mlflow-observability-secret -n <operator-ns> \
+#     --from-literal=MLFLOW_TRACING_ENABLED=true \
+#     --from-literal=MLFLOW_TRACKING_URI=https://mlflow.example.com \
+#     --from-literal=MLFLOW_TRACKING_AUTH=kubernetes-namespaced \
+#     --from-literal=MLFLOW_EXPERIMENT_NAME=ambient-code-sessions \
+#     --from-literal=MLFLOW_WORKSPACE=my-workspace \
+#     --from-literal=OBSERVABILITY_BACKENDS=langfuse,mlflow
+#
+# Keys:
+#   MLFLOW_TRACING_ENABLED  - "true" to enable secret copy + runner env injection
+#   MLFLOW_TRACKING_URI     - required on the runner for MLflow tracing
+#   MLFLOW_TRACKING_AUTH    - auth method; use "kubernetes-namespaced" for MLflow 3.11+ on K8s
+#                             so MLflow sends Authorization (service account JWT) and X-MLFLOW-WORKSPACE
+#   MLFLOW_EXPERIMENT_NAME  - optional (runner default: ambient-code-sessions)
+#   MLFLOW_WORKSPACE        - optional; fixed workspace id for X-MLFLOW-WORKSPACE instead of pod namespace
+#   OBSERVABILITY_BACKENDS  - optional; comma list: langfuse, mlflow (runner default: langfuse only if unset)
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ambient-admin-mlflow-observability-secret
+  namespace: CHANGE_ME_OPERATOR_NAMESPACE
+type: Opaque
+stringData:
+  MLFLOW_TRACING_ENABLED: "true"
+  MLFLOW_TRACKING_URI: "https://mlflow.example.com"
+  MLFLOW_TRACKING_AUTH: "kubernetes-namespaced"
+  MLFLOW_EXPERIMENT_NAME: "ambient-code-sessions"
+  OBSERVABILITY_BACKENDS: "langfuse,mlflow"

components/manifests/base/core/operator-deployment.yaml

@@ -99,6 +99,19 @@ spec:
               name: ambient-admin-langfuse-secret
               key: LANGFUSE_SECRET_KEY
               optional: true  # Optional: only needed if Langfuse enabled
+        # MLflow tracing (runner); optional secret managed by platform admin
+        - name: MLFLOW_TRACING_ENABLED
+          valueFrom:
+            secretKeyRef:
+              name: ambient-admin-mlflow-observability-secret
+              key: MLFLOW_TRACING_ENABLED
+              optional: true
+        - name: OBSERVABILITY_BACKENDS
+          valueFrom:
+            secretKeyRef:
+              name: ambient-admin-mlflow-observability-secret
+              key: OBSERVABILITY_BACKENDS
+              optional: true
         # Google OAuth client credentials for workspace-mcp
         - name: GOOGLE_OAUTH_CLIENT_ID
           valueFrom:

components/operator/internal/handlers/reconciler.go

@@ -115,8 +115,9 @@ func TransitionToStopped(ctx context.Context, session *unstructured.Unstructured
 	// Cleanup secrets
 	deleteCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
 	defer cancel()
-	_ = deleteAmbientVertexSecret(deleteCtx, namespace)
-	_ = deleteAmbientLangfuseSecret(deleteCtx, namespace)
+	_ = deleteAmbientVertexSecret(deleteCtx, namespace, name)
+	_ = deleteAmbientLangfuseSecret(deleteCtx, namespace, name)
+	_ = deleteAmbientMlflowObservabilitySecret(deleteCtx, namespace, name)
 
 	return nil
 }