bug: INFRAHUB_TRACE_INSECURE is ignored for the gRPC OTLP exporter (tracing fails with StatusCode.UNAVAILABLE)

[FragmentedPacket]

Summary

config.SETTINGS.trace.insecure (env INFRAHUB_TRACE_INSECURE, default True) is never
forwarded to the OpenTelemetry OTLP exporter. As a result the gRPC exporter falls back to
the SDK default (insecure=False) and attempts a TLS connection. Against a plaintext
OTLP collector (e.g. Grafana Tempo's OTLP/gRPC receiver on :4317), every span export
fails with:

Failed to export traces to <endpoint>, error code: StatusCode.UNAVAILABLE

No traces are ever delivered, even though INFRAHUB_TRACE_INSECURE=true is set.

Affected version

• Infrahub Enterprise 1.10.2 (also present in the current tracing code path)
• Exporter: opentelemetry-exporter-otlp-proto-grpc
• Collector under test: Grafana Tempo 2.10.3 (OTLP/gRPC, plaintext, :4317)

Root cause

backend/infrahub/trace.py builds the exporter without passing insecure:

elif exporter_protocol == "grpc":
    exporter = GRPCSpanExporter(endpoint=exporter_endpoint)   # insecure not passed

create_tracer_provider() / configure_trace() do not accept an insecure argument, and
the three callers never pass config.SETTINGS.trace.insecure:

• backend/infrahub/server.py:58
• backend/infrahub/cli/git_agent.py:108
• backend/infrahub/workers/infrahub_async.py:103

configure_trace(
    service="infrahub-server",
    version=__version__,
    exporter_type=config.SETTINGS.trace.exporter_type,
    exporter_endpoint=config.SETTINGS.trace.exporter_endpoint,
    exporter_protocol=config.SETTINGS.trace.exporter_protocol,
    # config.SETTINGS.trace.insecure is never passed
)

Meanwhile the setting is defined and defaults to True:

# backend/infrahub/config.py:1012
insecure: bool = Field(
    default=True,
    description="Use insecure connection (HTTP) if True, otherwise use secure connection (HTTPS)",
)

So the documented INFRAHUB_TRACE_INSECURE knob has no effect on the gRPC path.

Reproduction

With Infrahub configured for OTLP/gRPC to a plaintext collector:

INFRAHUB_TRACE_ENABLE=true
INFRAHUB_TRACE_EXPORTER_TYPE=otlp
INFRAHUB_TRACE_EXPORTER_PROTOCOL=grpc
INFRAHUB_TRACE_EXPORTER_ENDPOINT=<tempo-host>:4317
INFRAHUB_TRACE_INSECURE=true

Server logs stream StatusCode.UNAVAILABLE; the collector receives zero spans.

Minimal confirmation that the flag is the cause (same container, same endpoint):

Exporter construction	Result
GRPCSpanExporter(endpoint=EP) (current behavior, insecure=None → TLS)	SpanExportResult.FAILURE
GRPCSpanExporter(endpoint=EP, insecure=True)	SpanExportResult.SUCCESS
endpoint="http://"+EP (scheme forces insecure)	SpanExportResult.SUCCESS
env OTEL_EXPORTER_OTLP_TRACES_INSECURE=true	SpanExportResult.SUCCESS

Suggested fix

Thread insecure through and pass it to the gRPC exporter:

# trace.py
def create_tracer_provider(
    service: str,
    version: str,
    exporter_type: str,
    exporter_endpoint: str | None = None,
    exporter_protocol: str | None = None,
    insecure: bool = True,
) -> TracerProvider:
    ...
    elif exporter_protocol == "grpc":
        exporter = GRPCSpanExporter(endpoint=exporter_endpoint, insecure=insecure)

Add the same insecure parameter to configure_trace() and pass
config.SETTINGS.trace.insecure from all three callers.

Note: the OTLP HTTP exporter has no insecure argument — it derives TLS from the
endpoint URL scheme (http:// vs https://), so for http/protobuf the endpoint should
carry an explicit scheme. Worth handling/validating in the same change.

Workaround (no code change)

Set the standard OpenTelemetry env var, which the gRPC exporter honors directly:

OTEL_EXPORTER_OTLP_TRACES_INSECURE=true

(or give the endpoint an http:// scheme).

[FragmentedPacket]

/bug-analyze

[opsmill-bug-pipeline]

✅ Bug analyst agent completed successfully!

[opsmill-bug-pipeline]

Root cause analysis for #9869

Issue: bug: INFRAHUB_TRACE_INSECURE is ignored for the gRPC OTLP exporter (tracing fails with StatusCode.UNAVAILABLE)
Based on: afdb2f17ec3dd2d20447debdca8cea5a887e2590
Bug clarity: CLEAR
Code identification: RESOLVED

Root cause

The config.SETTINGS.trace.insecure setting is never threaded into create_tracer_provider()/configure_trace(), so the gRPC OTLP exporter is built without insecure=, falls back to the SDK default of TLS, and fails against a plaintext collector.

Affected files

• backend/infrahub/trace.py -- line 73: GRPCSpanExporter(endpoint=exporter_endpoint) is constructed with no insecure argument, so the SDK default (insecure=False → TLS) is always used.
• backend/infrahub/trace.py -- lines 57-63: create_tracer_provider() has no insecure parameter to forward.
• backend/infrahub/trace.py -- lines 97-103: configure_trace() has no insecure parameter and does not pass one down.
• backend/infrahub/server.py -- lines 58-64: caller omits insecure=config.SETTINGS.trace.insecure.
• backend/infrahub/cli/git_agent.py -- lines 108-114: caller omits insecure=config.SETTINGS.trace.insecure.
• backend/infrahub/workers/infrahub_async.py -- lines 103-109: caller omits insecure=config.SETTINGS.trace.insecure.
• backend/infrahub/config.py -- lines 1052-1054: TraceSettings.insecure is defined (default True, env INFRAHUB_TRACE_INSECURE) but is a dead knob — nothing reads it.

Explanation

The setting is correctly declared on TraceSettings (config.py:1049-1061) with insecure: bool = Field(default=True, ...) and env prefix INFRAHUB_TRACE_, so INFRAHUB_TRACE_INSECURE is parsed. But the value never leaves config: none of the three configure_trace(...) call sites pass it, and neither configure_trace() nor create_tracer_provider() accepts it. On the gRPC branch (trace.py:72-73) the exporter is therefore constructed as GRPCSpanExporter(endpoint=exporter_endpoint). The OTLP gRPC exporter's default when insecure is unset is a secure (TLS) channel, so exporting to a plaintext OTLP/gRPC receiver (e.g. Tempo on :4317) fails with StatusCode.UNAVAILABLE and no spans are delivered — exactly the reporter's observation and reproduction table.

The reporter's suggested fix (add insecure parameter, thread it through, pass insecure=insecure to GRPCSpanExporter) is correct and matches the SDK's supported constructor argument. Their env-var workaround (OTEL_EXPORTER_OTLP_TRACES_INSECURE=true, or an `(redacted) endpoint scheme) is also correct because the exporter honors that standard OTel env var directly.

Note on the HTTP path: the OTLP HTTP exporter (http/protobuf, trace.py:70-71) has no insecure constructor argument — it derives TLS from the endpoint URL scheme ((redacted) vs https://). So the insecureflag only meaningfully applies to the gRPC branch; the HTTP branch should not receive aninsecure` kwarg.

Fix strategy

Approach: Add an insecure: bool = True parameter to both create_tracer_provider() and configure_trace() in backend/infrahub/trace.py, have configure_trace() forward it to create_tracer_provider(), and pass insecure=insecure only into the GRPCSpanExporter(...) call on the grpc branch. Then update the three callers (server.py, cli/git_agent.py, workers/infrahub_async.py) to pass insecure=config.SETTINGS.trace.insecure alongside the existing exporter_* arguments. Default the new parameter to True to preserve the setting's documented default and avoid changing behavior for callers not updated in some other code path.

Scope: Small. Two signatures + one exporter call in trace.py, plus one added keyword argument at each of the three call sites. Roughly 8-12 changed lines total. Add a changelog fragment under changelog/ (bugfix). No config, GraphQL schema, DB, or auth changes.

Do NOT:

• Do NOT pass insecure= to the HTTPSpanExporter (http/protobuf) constructor — it does not accept that argument; TLS there is driven by the endpoint URL scheme. Only wire insecure into the gRPC branch.
• Do NOT read config.SETTINGS inside trace.py to grab the flag — keep trace.py free of the global settings dependency and let the callers pass the value, consistent with how exporter_type/exporter_endpoint/exporter_protocol are already threaded in.
• Do NOT invert the boolean — the config field means "insecure=True → plaintext", which maps directly to the exporter's insecure= kwarg. No negation.
• Do NOT set the OTel env var from code or introduce a new setting; the insecure field already exists.
• Do NOT refactor the surrounding tracing setup or the unrelated else/unbound-exporter edge case beyond what this fix requires.

Notes for downstream steps

• The cleanest test target is create_tracer_provider(...) (pure factory, no global state) rather than configure_trace() (which mutates the global tracer provider via trace.set_tracer_provider). Assert that when called with exporter_protocol="grpc" and insecure=<value>, the constructed GRPCSpanExporter receives that insecure value. The gRPC exporter stores channel config internally, so the most robust approach is to patch/spy on GRPCSpanExporter (the alias imported in trace.py) and assert the insecure kwarg it was called with — the test-writer should follow the repo's no-unittest.mock guidance where practical, but this is a thin third-party-constructor boundary; check dev/guidelines/backend/ and existing tests under backend/tests/ for the accepted pattern for asserting on third-party exporter construction.
• Verify the default remains True so behavior is unchanged when the flag is not explicitly set.
• Edge case to keep in mind (out of scope but nearby): in create_tracer_provider, if exporter_type == "otlp" and exporter_protocol is neither http/protobuf nor grpc, exporter is never assigned and raises UnboundLocalError. Do not fix it here unless it blocks the test; just don't regress it.
• TraceTransportProtocol / TraceExporterType are enums (config.py:1055-1060); the exporter branch compares against string values "grpc" / "http/protobuf", which works because the enum members are str-valued. No change needed there.

AGENT_ANALYSIS_COMPLETE

Generated by Bug analyst agent for #9869 · 89.6 AIC · ⌖ 29 AIC · ⊞ 11.5K · ◷
_{Comment /bug-analyze to run again}