Skip to content

fix(logs): staging fidelity fixes — tabs, newlines, Unicode positions, UTF-8 safety#49810

Draft
DDuongNguyen wants to merge 1 commit intojsaf/json-encode-strings-fixfrom
yoon/staging-fidelity-fixes
Draft

fix(logs): staging fidelity fixes — tabs, newlines, Unicode positions, UTF-8 safety#49810
DDuongNguyen wants to merge 1 commit intojsaf/json-encode-strings-fixfrom
yoon/staging-fidelity-fixes

Conversation

@DDuongNguyen
Copy link
Copy Markdown
Contributor

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

Copy link
Copy Markdown
Contributor Author

DDuongNguyen commented Apr 23, 2026

This was referenced Apr 23, 2026
@agent-platform-auto-pr
Copy link
Copy Markdown
Contributor

agent-platform-auto-pr Bot commented Apr 23, 2026

Go Package Import Differences

Baseline: a903409
Comparison: 85b5f31

binaryosarchchange
agentlinuxamd64
+11, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/google/flatbuffers/go
agentlinuxarm64
+11, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/google/flatbuffers/go
agentwindowsamd64
+11, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/google/flatbuffers/go
agentdarwinamd64
+11, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/google/flatbuffers/go
agentdarwinarm64
+11, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/google/flatbuffers/go
iot-agentlinuxamd64
+9, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
iot-agentlinuxarm64
+9, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
heroku-agentlinuxamd64
+11, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/google/flatbuffers/go
cluster-agentlinuxamd64
+9, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agentlinuxarm64
+9, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent-cloudfoundrylinuxamd64
+1, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent-cloudfoundrylinuxarm64
+1, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
dogstatsdlinuxamd64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
dogstatsdlinuxarm64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
process-agentlinuxamd64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
process-agentlinuxarm64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
process-agentwindowsamd64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
process-agentdarwinamd64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
process-agentdarwinarm64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
heroku-process-agentlinuxamd64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
security-agentlinuxamd64
+10, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/DataDog/datadog-agent/pkg/trace/log
security-agentlinuxarm64
+10, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/DataDog/datadog-agent/pkg/trace/log
security-agentwindowsamd64
+10, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/DataDog/datadog-agent/pkg/trace/log
system-probelinuxamd64
+10, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/DataDog/datadog-agent/pkg/trace/log
system-probelinuxarm64
+10, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/DataDog/datadog-agent/pkg/trace/log
system-probewindowsamd64
+10, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/DataDog/datadog-agent/pkg/trace/log
system-probedarwinamd64
+10, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/DataDog/datadog-agent/pkg/trace/log
system-probedarwinarm64
+10, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/DataDog/datadog-agent/pkg/trace/log
otel-agentlinuxamd64
+9, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
otel-agentlinuxarm64
+9, -0
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering
+github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging
+github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction
+github.com/DataDog/datadog-agent/pkg/logs/patterns/processor
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tags
+github.com/DataDog/datadog-agent/pkg/logs/patterns/token
+github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust
+github.com/DataDog/datadog-agent/pkg/logs/sender/grpc
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
privateactionrunnerlinuxamd64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
privateactionrunnerlinuxarm64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
privateactionrunnerwindowsamd64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
privateactionrunnerdarwinamd64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers
privateactionrunnerdarwinarm64
+2, -0
+github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
+github.com/planetscale/vtprotobuf/protohelpers

…, UTF-8 safety

Template reconstruction mismatches found in staging (flink-intakeshadow-metrics).

Pattern template fixes (pattern.go):
1. \t/\n/\r preserved — sanitizeForTemplateInto was stripping tab (0x09 < 0x20),
   newline, and carriage return. Fixes DNS records, ctr progress logs, ECR pulls,
   xDS ADS request bodies with pretty-printed JSON, curl output.
2. Trailing whitespace tokens no longer dropped — GetPatternString was skipping
   tokens where sanitizeForTemplateInto returned 0, silently losing trailing spaces.
3. Unicode rune positions — GetWildcardCharPositions emitted byte offsets; Java
   server expects UTF-16 code unit counts. Add sanitizeForTemplateRuneLen.

Proto safety (mock_state.go):
4. toValidUTF8 — proto3 string fields require valid UTF-8. Invalid bytes in
   wildcard values or raw log content would corrupt or silently drop datums.
   Applied in fillDynamicValue (string_value) and sendRawLog.

Preprocessor alignment (preprocessor/json.go):
5. JSONContextValues changed []string → []interface{} to match processor/json.go.
   Preserves native JSON types (numbers as numbers, not strings) for correct
   transport-layer encoding.

All fixes include regression tests using exact strings from production staging logs.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@TheSafo TheSafo force-pushed the yoon/staging-fidelity-fixes branch from 0165d0a to 85b5f31 Compare April 27, 2026 14:42
@TheSafo TheSafo force-pushed the graphite-base/49810 branch from 5d3e3e6 to 13c88a0 Compare April 27, 2026 14:42
@TheSafo TheSafo changed the base branch from graphite-base/49810 to jsaf/json-encode-strings-fix April 27, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant