Bound and checkpoint Datadog metric submissions#169
Open
Areson wants to merge 3 commits into
Open
Conversation
Size compressed metric payloads locally against both Datadog limits and submit the prepared bytes directly. Add Catalyst-shaped equivalence and benchmark coverage for the legacy and byte-budgeted paths. Co-authored-by: Codex <noreply@openai.com> Ai-assisted: true
Co-authored-by: Codex <noreply@openai.com> Ai-assisted: true
Co-authored-by: Codex <noreply@openai.com> Ai-assisted: true
Areson
marked this pull request as ready for review
July 18, 2026 21:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Large metric batches can exceed Datadog decompressed payload limits even when their compressed wire size is valid. Reactive 413 retries repeatedly rebuild oversized payloads, and retrying a partially accepted batch from the beginning can stall submission and duplicate acknowledged chunks.
What
Risk Assessment
Medium — this changes the production Datadog API submission path and Retry queue entries. Payloads are checked below both documented limits, 413 fallback remains bounded, DogStatsD behavior is preserved, and a one-time representative large-batch comparison confirmed decoded output parity; a request accepted by Datadog whose response is lost still cannot be made exactly-once without server-side idempotency.
References
A one-time local representative large-batch comparison reduced submission from 7.53s, 27 requests, and 22 HTTP 413 responses to 1.05s, 5 requests, and no HTTP 413 responses. The comparison harness was intentionally not included in this PR.
Validated with:
go test -mod=mod ./...go test -race ./sinkgo vet ./sinkGenerated with Codex