Add config schemas for comp/otelcol components#50408
Add config schemas for comp/otelcol components#50408gh-worker-dd-mergequeue-cf854d[bot] merged 6 commits intomainfrom
Conversation
truthbk
added
qa/no-code-change
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 89e6f9823f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| allOf: | ||
| - $ref: go.opentelemetry.io/collector/exporter/exporterhelper.timeout_config | ||
| - $ref: go.opentelemetry.io/collector/config/confighttp.client_config | ||
| - $ref: go.opentelemetry.io/collector/config/configtls.client_config |
There was a problem hiding this comment.
tls
This schema exposes configtls.client_config at the root via allOf, but ExporterConfig decodes that struct from mapstructure:"tls" (see comp/otelcol/otlp/components/exporter/serializerexporter/config.go), so TLS keys are expected under a nested tls: block. As written, schema-based validation can incorrectly accept top-level TLS keys (for example insecure) that the real config loader does not map the same way, which makes the schema inaccurate for downstream validation and editor tooling.
Useful? React with 👍 / 👎.
| allow_hostname_override: | ||
| type: boolean | ||
| cardinality: | ||
| $ref: /comp/core/tagger/types.tag_cardinality |
There was a problem hiding this comment.
$ref for cardinality
The new cardinality field points to /comp/core/tagger/types.tag_cardinality, but this repo does not provide a generated schema document for that target path, so consumers that resolve refs will hit a dangling local reference when validating infraattributesprocessor config. This breaks the stated goal of machine-validatable schemas for this component unless the referenced schema is also generated/published or the field is inlined.
Useful? React with 👍 / 👎.
|
🎯 Code Coverage (details) 🔗 Commit SHA: 9862911 | Docs | Datadog PR Page | Give us feedback! |
Files inventory check summaryFile checks results against ancestor eb7778b2: Results for datadog-agent_7.80.0~devel.git.527.9862911.pipeline.111949569-1_amd64.deb:No change detected |
Static quality checks✅ Please find below the results from static quality gates 32 successful checks with minimal change (< 2 KiB)
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 18ff3c1 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | -1.72 | [-4.61, +1.17] | 1 | Logs |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_metrics_logs | memory utilization | +1.24 | [+0.98, +1.50] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_logs | % cpu utilization | +0.57 | [-0.41, +1.54] | 1 | Logs bounds checks dashboard |
| ➖ | otlp_ingest_metrics | memory utilization | +0.38 | [+0.23, +0.53] | 1 | Logs |
| ➖ | ddot_metrics_sum_delta | memory utilization | +0.34 | [+0.16, +0.53] | 1 | Logs |
| ➖ | quality_gate_security_idle | memory utilization | +0.33 | [+0.26, +0.40] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_logs | memory utilization | +0.25 | [+0.19, +0.31] | 1 | Logs |
| ➖ | docker_containers_memory | memory utilization | +0.24 | [+0.14, +0.34] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | +0.14 | [+0.10, +0.19] | 1 | Logs bounds checks dashboard |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.03 | [-0.10, +0.15] | 1 | Logs |
| ➖ | quality_gate_security_no_fs_load | memory utilization | +0.02 | [-0.09, +0.12] | 1 | Logs bounds checks dashboard |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | +0.01 | [-0.46, +0.49] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.01 | [-0.19, +0.21] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | -0.00 | [-0.09, +0.09] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_v3 | ingress throughput | -0.01 | [-0.21, +0.20] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.04 | [-0.50, +0.43] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | -0.07 | [-0.46, +0.32] | 1 | Logs |
| ➖ | uds_dogstatsd_20mb_12k_contexts_20_senders | memory utilization | -0.07 | [-0.12, -0.02] | 1 | Logs |
| ➖ | otlp_ingest_logs | memory utilization | -0.08 | [-0.18, +0.01] | 1 | Logs |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | -0.10 | [-0.15, -0.06] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_metrics_sum_cumulativetodelta_exporter | memory utilization | -0.13 | [-0.37, +0.11] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.18 | [-0.36, -0.00] | 1 | Logs |
| ➖ | ddot_metrics | memory utilization | -0.36 | [-0.56, -0.17] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.46 | [-0.51, -0.42] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_metrics_sum_cumulative | memory utilization | -0.94 | [-1.10, -0.79] | 1 | Logs |
| ➖ | docker_containers_cpu | % cpu utilization | -1.72 | [-4.61, +1.17] | 1 | Logs |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | docker_containers_cpu | simple_check_run | 10/10 | 741 ≥ 26 | |
| ✅ | docker_containers_memory | memory_usage | 10/10 | 244.96MiB ≤ 370MiB | |
| ✅ | docker_containers_memory | simple_check_run | 10/10 | 550 ≥ 26 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | 0.16GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_0ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | 0.20GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_1000ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | 0.17GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_100ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | 0.18GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_500ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 142.27MiB ≤ 147MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 470.15MiB ≤ 495MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 175.28MiB ≤ 195MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 345.74 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 3 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 373.51MiB ≤ 430MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_security_idle | cpu_usage | 10/10 | 25.46 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_security_idle | memory_usage | 10/10 | 290.46MiB ≤ 330MiB | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | cpu_usage | 10/10 | 49.67 ≤ 70 | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | memory_usage | 10/10 | 267.67MiB ≤ 320MiB | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | cpu_usage | 10/10 | 18.87 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | memory_usage | 10/10 | 273.69MiB ≤ 320MiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
| kubernetes_kubelet_host: | ||
| description: Kubernetes / kubelet settings for K8s tag enrichment (standalone mode). When the otel-agent runs on a Kubernetes node these allow the workloadmeta kubelet collector and the local tagger to reach the kubelet API without needing a separate datadog.yaml. | ||
| type: string | ||
| metadata_interval: |
There was a problem hiding this comment.
I'm not yet familiar with used schema generation tool like cmd/schemagen
Just noticed that fields specified in schema has no comments, while in source code there are:
| MetadataInterval | // seconds; 0 = use agent default (1800) | no description |
|---|---|---|
| TaggerServerPort | // 0 = auto-assign | no description |
| TaggerServerAddr | // Default: localhost | no description |
| TaggerMaxMessageSize | // Default: 4MB | no description |
| TaggerMaxConcurrentSync | // Default: 5 | no description |
| SecretBackendTimeout | // seconds; 0 = agent default (30s) | no description |
| SecretBackendOutputMaxSize | // bytes; 0 = agent default | no description |
unf. schemagen reads field.Doc only — that's the Go AST's doc comment (the comment block above the field). It does not read field.Comment, which is the inline/trailing comment on the same line.
So for code like this:
MetadataInterval int
mapstructure:"metadata_interval"// seconds; 0 = use agent default (1800)
The // seconds; 0 = ... is field.Comment (inline), not field.Doc — schemagen ignores it entirely. But if it were written as:
// MetadataInterval in seconds. 0 = use agent default (1800).
MetadataInterval intmapstructure:"metadata_interval"
That becomes field.Doc and schemagen would pick it up.
There was a problem hiding this comment.
I didn't really want to make changes to the go code here; but you're right, it's a good opportunity to allow the tool to pick up on this valuable configuration context.
Introduce upstream-style config.schema.yaml files for OTel Collector
components living under comp/otelcol/, mirroring the format used in
opentelemetry-collector-contrib (JSON Schema serialised as YAML, with
$defs/$ref/x-pointer/x-optional extensions).
Components covered:
- exporter/serializerexporter
- exporter/logsagentexporter
- processor/infraattributesprocessor
- ddflareextension (hand-written; schemagen does not support
embedded *pointer fields with mapstructure:",squash")
- ddprofilingextension
- dogtelextension
Schemas were generated using upstream's cmd/schemagen tool, with one
exception noted above. A metadata.yaml is added to each component dir
so schemagen can be re-run, and a comp/otelcol/.schemagen.yaml settings
file declares the namespace, type mappings, and allowed external $refs
(go.opentelemetry.io/collector, opentelemetry-collector-contrib,
DataDog/datadog-agent).
For datadogexporter, which has no local Config struct (it accepts
upstream's pkg/datadog/config.Config), a README is added pointing to
the upstream schema as the source of truth.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- serializerexporter schema: move configtls.client_config from root
allOf to properties.tls. Schemagen incorrectly squashes every Go-
embedded field, ignoring the explicit mapstructure:"tls" tag.
- ddprofilingextension: fix copy-pasted Endpoint comment ("Default:
BuildInfo.Version" belongs to Version, not Endpoint). The field is
the local port the profiling HTTP server listens on; default "7501".
Regenerate the schema to pick up the corrected description.
- logsagentexporter Config: add mapstructure:"-" to OtelSource,
LogSourceName, and OrchestratorConfig. These are runtime-injected
(constants set at factory wiring; OrchestratorConfig.Hostname is an
Fx Component) and were being silently bound from lowercased YAML
keys, drifting from the schema. Inner OrchestratorConfig fields
remain untagged: closed at the parent and may yet be exposed
to users in the future.
- metadata.yaml across all 6 components: add display_name and drop
empty distributions: [] to match upstream conventions.
- ddprofilingextension metadata.yaml: align stability with factory.go,
which returns component.StabilityLevelDevelopment, not Beta.
- .schemagen.yaml: add a header comment documenting how to regenerate
schemas (build command, invocation, walk-up lookup) and the two
known schemagen quirks that require hand-tuning (embedded pointer
fields and embedded fields with non-squash mapstructure keys).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The generated config.schema.yaml referenced /comp/core/tagger/types.tag_cardinality, but no schema is published for that package and we don't want to publish one (it has many unrelated types). Schemagen emitted the ref because github.com/DataDog/datadog-agent is in allowedRefs. Map types.TagCardinality to a primitive integer in .schemagen.yaml so the schema mirrors the YAML decode behaviour today (TagCardinality is a typed int with iota constants; there is no UnmarshalText hook). Add a Go doc-comment on Config.Cardinality enumerating the accepted values (0=low, 1=orchestrator, 2=high), which schemagen now picks up as the field description so it survives regeneration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Schemagen reads field.Doc (leading comments) but ignores field.Comment
(trailing inline comments), so per-field annotations like
"// seconds; 0 = use agent default (1800)" were invisible to the
generator. The previous schema also misplaced group-header descriptions
because schemagen alphabetises fields, leaving most properties without
any description and the headers ("Tagger server settings", etc.)
attached to the wrong field.
Rewrite each field's documentation as a self-contained leading godoc
comment so schemagen picks it up and group-ordering no longer matters.
The regenerated schema now has a description on every property.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This reverts commit 245086a.
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
3 participants
What does this PR do?
Adds upstream-style
config.schema.yamlfiles for the OpenTelemetry Collector components living undercomp/otelcol/, mirroring the JSON-Schema-as-YAML format used byopentelemetry-collector-contrib(with$defs/$ref/x-pointer/x-optionalextensions).Components covered:
comp/otelcol/otlp/components/exporter/serializerexportercomp/otelcol/otlp/components/exporter/logsagentexportercomp/otelcol/otlp/components/processor/infraattributesprocessorcomp/otelcol/ddflareextension/implcomp/otelcol/ddprofilingextension/implcomp/otelcol/dogtelextension/implFor
comp/otelcol/otlp/components/exporter/datadogexporter, which has no localConfigstruct and consumes the upstreampkg/datadog/config.Configdirectly, aREADME.mdis added pointing to the upstream schema as the source of truth.A
metadata.yamlis also added next to each component'sconfig.goso the generator can be re-run, and acomp/otelcol/.schemagen.yamlsettings file declares ournamespace, type mappings (time.Duration→string/format: duration, etc.) andallowedRefs(go.opentelemetry.io/collector,opentelemetry-collector-contrib,DataDog/datadog-agent).Motivation
Provide machine-readable, human-readable specifications of the YAML configuration accepted by each Datadog-authored OTel Collector component. This unblocks downstream tooling (validation, documentation generation, IDE support) and keeps the format in lockstep with how upstream OTel components advertise their config surface.
Describe how you validated your changes
cmd/schemagentool, with the settings file atcomp/otelcol/.schemagen.yaml.yaml.safe_load(Python).\$reftargets against each component'sConfigstruct.schemagenre-runs from the new settings location produce byte-identical output for an already-generated component.Additional Notes
ddflareextension's schema is hand-written because schemagen'saddEmbeddedFieldonly handles*ast.Identand*ast.SelectorExpr— it errors on*ast.StarExpr(i.e. embedded pointer fields withmapstructure:\",squash\"). The localConfigembeds*confighttp.ServerConfigin that shape. The four-line schema\$refsgo.opentelemetry.io/collector/config/confighttp.server_configviaallOf. A small upstream patch to schemagen would let us regenerate this in future.\$refpaths to upstream OTel types use the fully qualified form (e.g.github.com/open-telemetry/opentelemetry-collector-contrib/pkg/datadog/config.api_config) rather than the slash-prefixed form upstream uses internally. This is correct: from this repo's namespace those types are external, and schemagen produces the full path automatically. Both forms are valid JSON Schema URI references per RFC 3986.metricsclient/,statsprocessor/, and the now-emptyconnector/datadogconnector/directory are intentionally not covered: the first two are internal helper libraries (nocomponent.Config), and the third no longer hosts a registered component (moved upstream in DDOT imports datadogconnector from OSS #43479; only a stalemodule_test_output.jsonremains).mapstructuretags are correctly omitted from the generated schemas (per schemagen's contract). Forlogsagentexporterthis meansOtelSource,LogSourceName, andOrchestratorConfigdo not appear — they are runtime-injected, not user-configurable via YAML.