Skip to content

fix(observability): give Substrate's OTLP export a tenant and a way out - #638

Merged
QuentinBisson merged 1 commit into
mainfrom
fix/substrate-otlp-tenant
Sep 24, 2026
Merged

QuentinBisson merged 1 commit into
mainfrom
fix/substrate-otlp-tenant

Conversation

@QuentinBisson

@QuentinBisson QuentinBisson commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On gazelle, ate-api-server, ate-controller, atelet and atenet-router export to otlp-gateway.kube-system.svc:4317, and Tempo has no span from them (giantswarm/giantswarm#36711). No Substrate pod carries observability.giantswarm.io/tenant, and the chart has no headers key, so otlp-gateway drops every signal as tenant-less. Also, only the ate-api-server policy allows 4317, so the other three exports never reach the collector.

Change

  • substrate.podLabels: {observability.giantswarm.io/tenant: giantswarm} in the meta chart. The substrate chart takes the key from 1.1.0 (feat(chart): podLabels on the chart's pods substrate#55, which carries feat(helm): add podLabels for the chart's pods kagent-dev/substrate#48). 1.0.x ignores it, so the label arrives with the re-pin's range move, not with this PR.
  • agent-platform.substrate.otlpEgress opens the namespace and port of each enabled signal's endpoint on the four exporters' Cilium policies. It replaces ate-api-server's fixed cluster:4317 rule, with a golden hold in verify-target.py.
  • Sampling stays at parentbased_traceidratio 0.01. Spans under a kagent turn follow the turn's decision, so the ratio applies only to parentless background work.
  • make verify-substrate-otlp.

Checklist

  • Update changelog in CHANGELOG.md.
  • Make sure values.yaml and values.schema.json are valid.

@circleci-architect

circleci-architect Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Published Helm chart agent-platform-connectivity

4.66.8-r72dfe194t20260924110036h00c0f9e
Chart agent-platform-connectivity
Version 4.66.8-r72dfe194t20260924110036h00c0f9e
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.66.8-r72dfe194t20260924110036h00c0f9e
Digest sha256:4b6f79e6a4cf93424babef5120022e6ca795a63099c015e8dbb4370ebbea1a0b
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.66.8-r72dfe194t20260924110036h00c0f9e

Posted by architect-orb · build 12661 · commit 00c0f9e · updated in place on every push

@circleci-architect

circleci-architect Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Published Helm chart agent-platform

4.66.8-r72dfe194t20260924110036h00c0f9e
Chart agent-platform
Version 4.66.8-r72dfe194t20260924110036h00c0f9e
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.66.8-r72dfe194t20260924110036h00c0f9e
Digest sha256:2057afca72e37404efc109b8f0dc40963e0ff91abd868a87b851bcbbd9aeb65d
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.66.8-r72dfe194t20260924110036h00c0f9e

Posted by architect-orb · build 12662 · commit 00c0f9e · updated in place on every push

The meta chart sets substrate.podLabels to the tenant label otlp-gateway
routes a headerless export by, and the connectivity chart's policies of
the four exporters open the endpoint substrate.otel names instead of a
fixed cluster:4317 rule on ate-api-server alone.
@QuentinBisson
QuentinBisson force-pushed the fix/substrate-otlp-tenant branch from fbb0b57 to 00c0f9e Compare September 24, 2026 11:03
@QuentinBisson
QuentinBisson marked this pull request as ready for review September 24, 2026 11:08
@QuentinBisson
QuentinBisson requested a review from a team as a code owner September 24, 2026 11:08
@QuentinBisson
QuentinBisson merged commit c29c06b into main Sep 24, 2026
13 checks passed
@QuentinBisson
QuentinBisson deleted the fix/substrate-otlp-tenant branch September 24, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant