feat(logs): max template size limit + json_as_raw raw bypass#49552
Draft
DDuongNguyen wants to merge 2 commits intojsaf/evict-correctlyfrom
Draft
feat(logs): max template size limit + json_as_raw raw bypass#49552DDuongNguyen wants to merge 2 commits intojsaf/evict-correctlyfrom
DDuongNguyen wants to merge 2 commits intojsaf/evict-correctlyfrom
Conversation
This was referenced Apr 17, 2026
Contributor
Author
Contributor
Go Package Import DifferencesBaseline: a903409
|
Contributor
Files inventory check summaryFile checks results against ancestor a9034095: Results for datadog-agent_7.79.0~devel.git.459.3865993.pipeline.108330740-1_amd64.deb:Detected file changes:
|
Two safety valves that prevent stateful encoding state bloat: **logs_config.patterns.max_template_bytes (default: 1024)** Rejects logs whose raw token content exceeds this byte threshold, sending them as RawLog datums instead of creating pattern state. Fixes snapshot bloat from huge one-off logs (e.g. AWS instance metadata dumps ~900KB each → ~8MB snapshot overhead from 9 patterns). Returns PatternTooLarge from ClusterManager.Add so the caller never dereferences a nil pattern. **logs_config.patterns.json_as_raw (default: false)** When true, JSON logs bypass stateful encoding entirely (no preprocessing, no tokenization, no clustering) and are sent as RawLog datums. Eliminates PreprocessJSON + tokenization + clustering cost for JSON-heavy workloads. Trade-off: no pattern compression; transport-level compression still applies. Useful for A/B testing CPU vs wire efficiency in staging. Both features route to the existing sendRawLog path in both the batch pipeline (processBatch) and single-message path (processMessage). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
DDuongNguyen
force-pushed
the
yoon/max-template-raw-bypass
branch
from
f825ba6 to
3865993
Compare
UseNumber in json.Unmarshal so numbers stay as json.Number (string) instead of float64. Without this, integers larger than 2^53 (trace IDs, span IDs, snowflake IDs) silently lose precision through the unmarshal/marshal round-trip — e.g. 9999999999999999 becomes 10000000000000000.
This was referenced Apr 24, 2026
Draft
Draft
Draft
Draft
This was referenced May 4, 2026
Draft
Draft
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Motivation
Describe how you validated your changes
Additional Notes