support metrics v3 in fakeintake#51150
Conversation
|
@codex review |
dd-octo-sts
Bot
added
internal
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69873fff60
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| "/api/v1/check_run": getCheckRunPayLoadJSON, | ||
| "/api/v1/connections": getConnectionsPayLoadProtobuf, | ||
| "/api/beta/sketches": getSketchPayloadProtobuf, | ||
| "/api/intake/metrics/v3/series": getMetricV3SeriesPayload, |
There was a problem hiding this comment.
This change enables serializer_experimental_use_v3_api.sketches.endpoints (host and Helm paths), so when V3 is turned on the agent sends sketch payloads to /api/intake/metrics/v3/sketches; however fakeintake only adds a parser for /api/intake/metrics/v3/series here, and FilterSketches() still reads /api/beta/sketches. In V3 mode, distribution/sketch assertions will therefore silently see no data even though payloads are being sent, which breaks end-to-end metric coverage for sketches.
Useful? React with 👍 / 👎.
Files inventory check summaryFile checks results against ancestor faf049aa: Results for datadog-agent_7.81.0~devel.git.368.4f0c860.pipeline.116250035-1_amd64.deb:No change detected |
atanzu
force-pushed
the
mark.kirichenko/support-v3-in-fakeintake
branch
from
69873ff to
46d4a45
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 46d4a4504b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| "/api/v1/check_run": getCheckRunPayLoadJSON, | ||
| "/api/v1/connections": getConnectionsPayLoadProtobuf, | ||
| "/api/beta/sketches": getSketchPayloadProtobuf, | ||
| "/api/intake/metrics/v3/series": getMetricV3SeriesPayload, |
There was a problem hiding this comment.
This map adds /api/intake/metrics/v3/series but leaves out /api/intake/metrics/v3/sketches, even though WithV3MetricsEnabled now configures both series and sketches V3 endpoints. In that configuration, sketch payloads are sent to the V3 sketches route (comp/forwarder/defaultforwarder/endpoints/endpoints.go) and fakeintake won’t decode them (and FilterSketches still reads /api/beta/sketches), so sketch-based assertions under V3 mode silently miss data. Either add V3-sketch parsing/client support or stop enabling the sketches V3 endpoint until that support exists.
Useful? React with 👍 / 👎.
atanzu
changed the title
atanzu
force-pushed
the
mark.kirichenko/support-v3-in-fakeintake
branch
from
46d4a45 to
b2b4d07
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b2b4d07492
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| "/api/v1/check_run": getCheckRunPayLoadJSON, | ||
| "/api/v1/connections": getConnectionsPayLoadProtobuf, | ||
| "/api/beta/sketches": getSketchPayloadProtobuf, | ||
| "/api/intake/metrics/v3/series": getMetricV3SeriesPayload, |
There was a problem hiding this comment.
When V3 is enabled, the new config paths set both series and sketches V3 endpoints to fakeintake, so sketch payloads are sent to /api/intake/metrics/v3/sketches. This parser map only adds /api/intake/metrics/v3/series, so V3 sketch payloads remain unparsed and FilterSketches() (which still reads /api/beta/sketches) will silently miss them. Any V3-enabled tests asserting distributions/sketches can therefore fail or give false negatives.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I don't know if we have tests using sketches but I think it's worth a check?
There was a problem hiding this comment.
I agree with @vickenty :
We can leave sketches out entirely from the config, or separate them from series so we can enable it without sketches.
But within the scope of this PR I believe it is OK.
atanzu
force-pushed
the
mark.kirichenko/support-v3-in-fakeintake
branch
from
b2b4d07 to
8995a94
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8995a94b17
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| sketches: | ||
| endpoints: | ||
| - %[3]s://%[1]s:%[2]d |
There was a problem hiding this comment.
This option now enables serializer_experimental_use_v3_api.sketches.endpoints, which redirects sketch/distribution traffic to /api/intake/metrics/v3/sketches, but fakeintake still has no corresponding v3-sketch retrieval path (FilterSketches still queries /api/beta/sketches, and there is no /api/intake/metrics/v3/sketches handler in serverstore/parser.go). As soon as a v3-enabled test needs to assert sketch metrics, it will get empty results despite sketches being sent.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
We can leave sketches out entirely from the config, or separate them from series so we can enable it without sketches.
atanzu
force-pushed
the
mark.kirichenko/support-v3-in-fakeintake
branch
from
8995a94 to
ca9624c
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ca9624c4fe
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| sketches: | ||
| endpoints: | ||
| - %[3]s://%[1]s:%[2]d |
There was a problem hiding this comment.
This now configures serializer_experimental_use_v3_api.sketches.endpoints, so sketch/distribution payloads are sent to /api/intake/metrics/v3/sketches, but fakeintake still only exposes sketch parsing/querying on /api/beta/sketches (FilterSketches/GetSketchNames in test/fakeintake/client/client.go). In suites that enable V3 and assert on sketch metrics, those assertions will incorrectly see no data because sketches have been rerouted to an unsupported endpoint.
Useful? React with 👍 / 👎.
atanzu
force-pushed
the
mark.kirichenko/support-v3-in-fakeintake
branch
from
ca9624c to
3655bb2
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3655bb2f3e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| sketches: | ||
| endpoints: | ||
| - %[3]s://%[1]s:%[2]d |
There was a problem hiding this comment.
WithV3MetricsEnabled also rewrites sketch traffic to V3, but this change set only adds fakeintake support for /api/intake/metrics/v3/series (new parser/client path) and adds no parser/client support for /api/intake/metrics/v3/sketches. In E2E runs that emit distribution/sketch metrics with V3 enabled, sketch payloads are sent to an endpoint fakeintake cannot decode/query, so FilterSketches-style assertions will silently see no data and tests can miss or misreport regressions.
Useful? React with 👍 / 👎.
atanzu
added
the
qa/done
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3655bb2f3e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| } | ||
| envVars := pulumi.Map{ | ||
| "DD_SERIALIZER_EXPERIMENTAL_USE_V3_API_SERIES_ENDPOINTS": pulumi.Sprintf("%s", p.intakeURL), | ||
| "DD_SERIALIZER_EXPERIMENTAL_USE_V3_API_SKETCHES_ENDPOINTS": pulumi.Sprintf("%s", p.intakeURL), |
There was a problem hiding this comment.
WithV3MetricsEnabled now sets DD_SERIALIZER_EXPERIMENTAL_USE_V3_API_SKETCHES_ENDPOINTS, which moves distribution/sketch traffic to /api/intake/metrics/v3/sketches, but this change only adds V3 handling for /api/intake/metrics/v3/series (test/fakeintake/server/serverstore/parser.go) and the client still reads sketches from /api/beta/sketches (test/fakeintake/client/client.go). In V3-enabled E2E runs that assert sketch/distribution metrics, fakeintake will silently return no sketches even though the agent is sending them, causing false negatives.
Useful? React with 👍 / 👎.
| sketches: | ||
| endpoints: | ||
| - %[3]s://%[1]s:%[2]d |
There was a problem hiding this comment.
We can leave sketches out entirely from the config, or separate them from series so we can enable it without sketches.
| if p.intakeHostname == nil || p.intakePort == nil || p.intakeScheme == nil { | ||
| return fmt.Errorf("WithV3MetricsEnabled must be called after WithFakeintake or WithIntakeHostname") | ||
| } | ||
| v3Config := pulumi.Sprintf(`serializer_experimental_use_v3_api: |
There was a problem hiding this comment.
This is my personal preference, but I try to avoid plain text templating like this. I'd make a struct and serialize it with proper serializer, just to avoid potential issues with quoting strings or counting spaces.
There was a problem hiding this comment.
I tried to play with that in rev2 but not sure if it has become significantly better.
| // Store so that WithV3MetricsEnabled (applied after) can build V3 endpoints config. | ||
| p.intakeScheme = scheme | ||
| p.intakeHostname = hostname | ||
| p.intakePort = port |
There was a problem hiding this comment.
With this, we format the url twice. Would it make sense to construct it here, stash it and use the complete string in two places?
There was a problem hiding this comment.
Now that we save the URL, do we still need to store intakeScheme and friends?
| return nil, fmt.Errorf("v3 next sketch point: %w", err) | ||
| } | ||
| } | ||
| continue |
There was a problem hiding this comment.
We should return an error if a sketch is encountered in a series payload, it would be a serious bug in the agent.
| case encodingDeflate: | ||
| rc, err = zlib.NewReader(bytes.NewReader(payload)) | ||
| case encodingZstd: | ||
| rc = zstd.NewReader(bytes.NewReader(payload)) |
There was a problem hiding this comment.
This should work with zstd with the latest head of DataDog/zstd (v1.5.8-0.20250922095318-5c504fb5d923).
There was a problem hiding this comment.
The problem is, I don't see a stable 1.5.8 tag. Would that be OK to have that specific pinned version?
There was a problem hiding this comment.
I'll follow up on the release, but yes, it should be OK to pin pre-release version.
There was a problem hiding this comment.
It was released as 1.5.7+patch1, +patch is how we version our changes on top of the upstream.
| // MetricSeriesV3 represents a single time series decoded from a V3 metrics intake payload | ||
| // (/api/intake/metrics/v3/series). The V3 format is a column-oriented protobuf encoding | ||
| // described in pkg/proto/datadog/dogstatsdhttp/payload.proto. | ||
| type MetricSeriesV3 struct { |
There was a problem hiding this comment.
Would it make sense to use the old metricAggregator for v3 as well? V3 is expected to be equivalent (at least until we turn on sketches), and should be transparent change. Ideally, the existing tests that currently use v2 protocol should upgrade when we enable v3 by default (and it would be nice if we can run the same suite in parallel for both versions). WDYT?
There was a problem hiding this comment.
My initial intent was to make this check as explicit as possible. You are right in terms of usage of a single metricAggregator and its transparency: after the flip the modified fakeintake shall work as is. However, for now we still need to have a clear vision between v2 and v3 (and check that we don't have mixed traffic), so I will update a couple of the existing test suites to have a v3 variant.
There was a problem hiding this comment.
Yep, with some intermediate glue we can process both v2 and v3 transparently. Still, I've added explicit test cases to know exactly which protocol do we use.
atanzu
force-pushed
the
mark.kirichenko/support-v3-in-fakeintake
branch
from
3655bb2 to
07cb2cd
Compare
Static quality checks✅ Please find below the results from static quality gates Successful checksInfo
18 successful checks with minimal change (< 2 KiB)
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: faf049a Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +0.03 | [-2.90, +2.97] | 1 | Logs |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_metrics_logs | memory utilization | +3.42 | [+3.15, +3.69] | 1 | Logs bounds checks dashboard |
| ➖ | otlp_ingest_metrics | memory utilization | +2.03 | [+1.87, +2.19] | 1 | Logs |
| ➖ | otlp_ingest_logs | memory utilization | +1.24 | [+1.15, +1.34] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | +0.65 | [+0.46, +0.83] | 1 | Logs |
| ➖ | ddot_metrics_sum_delta | memory utilization | +0.51 | [+0.34, +0.69] | 1 | Logs |
| ➖ | uds_dogstatsd_20mb_12k_contexts_20_senders | memory utilization | +0.50 | [+0.45, +0.55] | 1 | Logs |
| ➖ | ddot_metrics | memory utilization | +0.45 | [+0.25, +0.65] | 1 | Logs |
| ➖ | file_tree | memory utilization | +0.43 | [+0.38, +0.48] | 1 | Logs |
| ➖ | ddot_logs | memory utilization | +0.25 | [+0.19, +0.31] | 1 | Logs |
| ➖ | docker_containers_memory | memory utilization | +0.13 | [+0.03, +0.23] | 1 | Logs |
| ➖ | quality_gate_logs | % cpu utilization | +0.10 | [-0.94, +1.13] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | +0.07 | [+0.02, +0.12] | 1 | Logs bounds checks dashboard |
| ➖ | docker_containers_cpu | % cpu utilization | +0.03 | [-2.90, +2.97] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | +0.02 | [-0.42, +0.46] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.01 | [-0.20, +0.22] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.01 | [-0.08, +0.10] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_v3 | ingress throughput | -0.00 | [-0.20, +0.20] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | -0.01 | [-0.42, +0.39] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | -0.05 | [-0.19, +0.10] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | -0.06 | [-0.54, +0.42] | 1 | Logs |
| ➖ | ddot_metrics_sum_cumulativetodelta_exporter | memory utilization | -0.19 | [-0.43, +0.05] | 1 | Logs |
| ➖ | ddot_metrics_sum_cumulative | memory utilization | -0.23 | [-0.39, -0.08] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.34 | [-0.37, -0.30] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | docker_containers_cpu | simple_check_run | 10/10 | 725 ≥ 26 | |
| ✅ | docker_containers_memory | memory_usage | 10/10 | 246.29MiB ≤ 370MiB | |
| ✅ | docker_containers_memory | simple_check_run | 10/10 | 697 ≥ 26 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | 0.16GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_0ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | 0.20GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_1000ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | 0.17GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_100ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | 0.18GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_500ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 144.09MiB ≤ 147MiB | bounds checks dashboard |
| ✅ | quality_gate_idle | total_bytes_received | 10/10 | 740.76KiB ≤ 819.20KiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 475.09MiB ≤ 495MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | total_bytes_received | 10/10 | 1.14MiB ≤ 1.25MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 175.48MiB ≤ 195MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_logs | total_bytes_received | 10/10 | 264.33MiB ≤ 292MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 359.19 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 418.49MiB ≤ 430MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | total_bytes_received | 10/10 | 0.93GiB ≤ 1.04GiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
atanzu
force-pushed
the
mark.kirichenko/support-v3-in-fakeintake
branch
2 times, most recently
from
60b632e to
9b358c7
Compare
chouetz
changed the title
chouetz
left a comment
chouetz
left a comment
There was a problem hiding this comment.
Might deserve a double check on the last codex comment, I assume it's not a blocker though.
| "/api/v1/check_run": getCheckRunPayLoadJSON, | ||
| "/api/v1/connections": getConnectionsPayLoadProtobuf, | ||
| "/api/beta/sketches": getSketchPayloadProtobuf, | ||
| "/api/intake/metrics/v3/series": getMetricV3SeriesPayload, |
There was a problem hiding this comment.
I don't know if we have tests using sketches but I think it's worth a check?
| // Store so that WithV3MetricsEnabled (applied after) can build V3 endpoints config. | ||
| p.intakeScheme = scheme | ||
| p.intakeHostname = hostname | ||
| p.intakePort = port |
There was a problem hiding this comment.
Now that we save the URL, do we still need to store intakeScheme and friends?
| // Only series are redirected; sketches V3 support is not yet implemented in fakeintake. | ||
| // | ||
| // Must be called after WithFakeintake (or WithIntakeHostname) so the intake URL is known. | ||
| func WithV3MetricsEnabled() func(*Params) error { |
There was a problem hiding this comment.
Would it make sense to add this option to the list in the comment attached to Params?
There was a problem hiding this comment.
I wanted to make this part as explicit as possible.
There was a problem hiding this comment.
What part? What I meant was to add this method the list of available options.
There was a problem hiding this comment.
Ah, just list it there. Ack. I misunderstood your initial comment: I thought it was about enabling v3 in some other, more implicit way.
| // Only series are redirected; sketches V3 support is not yet implemented in fakeintake. | ||
| // | ||
| // Must be called after WithFakeintake or WithIntake so the intake URL is known. | ||
| func WithV3MetricsEnabled() func(*Params) error { |
There was a problem hiding this comment.
Same here, would it make sense to add this option to the list in the comment attached to Params?
| "slices" | ||
|
|
||
| agentpayload "github.com/DataDog/agent-payload/v5/gogen" | ||
| pb "github.com/DataDog/datadog-agent/pkg/proto/pbgo/dogstatsdhttp" |
There was a problem hiding this comment.
Let's use this instead definition instead, that repo should contain compiled version as well. It includes units, so we won't need to parse them from bytes.
There was a problem hiding this comment.
That's a nice one, thanks.
There was a problem hiding this comment.
There is no compiled version for v3, but that's not a blocker.
| // // Acessors for metric data point can be called. | ||
| // } | ||
| // } | ||
| type MetricDataReader struct { |
There was a problem hiding this comment.
Together with the schema, let's copy the implementation from the intake instead to use with the pb file from agent-payload. It covers top-level metadata, and it would make it easier to maintain in the future.
atanzu
force-pushed
the
mark.kirichenko/support-v3-in-fakeintake
branch
4 times, most recently
from
5e9db01 to
434fdc6
Compare
Support decoding of V3 payload in fakeintake, so we could test the new payload type. We use the intake's reader implementation to keep the code base consistent. Signed-off-by: Mark Kirichenko <mark.kirichenko@datadoghq.com>
Add V3 protocol support and create an explicit test case to verify that we can operate in V2- or V3-only modes where all the metrics traffic reaches the expected endpoint. Signed-off-by: Mark Kirichenko <mark.kirichenko@datadoghq.com>
Add a test to verify that we can send OTel traffic via V3 protocol when we configure the Agent to use V3. Signed-off-by: Mark Kirichenko <mark.kirichenko@datadoghq.com>
atanzu
force-pushed
the
mark.kirichenko/support-v3-in-fakeintake
branch
from
9343df1 to
ce03e7e
Compare
Signed-off-by: Mark Kirichenko <mark.kirichenko@datadoghq.com>
atanzu
force-pushed
the
mark.kirichenko/support-v3-in-fakeintake
branch
from
ce03e7e to
4f0c860
Compare
6 participants
What does this PR do?
Allow us to test V3 payloads with fakeintake.
Motivation
Current e2e test setup doesn't support the V3 payload format. We want to address this gap.
We add V3 metrics intake support, which relies on the V3 parser from the intake. We cannot import this module directly, so we use a copy of it.
Describe how you validated your changes
We add new e2e tests for our usual metrics and for OTel data which verify that we can send data to a V3 endpoint instead of V2.
Additional Notes
V3 payload parser is a copy from the intake, so we need to update both in parallel in case of any changes.