json presence

[TheSafo]

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

[TheSafo]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• json presence #50402 👈 (View in Graphite)
• jsaf/flatten-json-encoding-values #50349
• extedned json nesting #50347
• flat-log-encoding #50321
• fix delta encoding #50133
• ttl in config #50121
• lazy snapshot sending #50120
• tag manager addition #50049
• don't send deletes, tighten TTL #50047
• utf8 #49898 : 2 other dependent PRs (#49961 , #50036 )
• fix(logs): staging fidelity fixes — tabs, newlines, Unicode positions, UTF-8 safety #49810
• better json #49672
• local proto plus servic #49628
• feat(logs): periodically remove unused entry from dict #49494
• feat(logs): max template size limit + json_as_raw raw bypass #49552
• fix(logs): send delete on tag eviction #49493
• fix(logs): rebuild libpatterns.so targeting glibc 2.30 (was 2.34) #49446
• feat(logs): send JSON via dict-encoded sets of keys #49404
• feat(logs): add dual-send shadow gRPC pipeline #49327
• feat(logs): wire stateful encoding into log agent pipeline #49220
• feat(logs): add pattern extraction to gRPC stateful sender #49219
• feat(logs): add pattern clustering, tag management, and JSON processor #49218
• feat(logs): add generic eviction framework for pattern management #49217
• feat(logs): add Rust FFI tokenizer bridge with FlatBuffers #49216
• feat(logs): add stateful encoding module plumbing and token types #49215
• [Logs Stateful Encoding] transport layer initial implementation #48879
• logs-stateful-main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[agent-platform-auto-pr]

Go Package Import Differences

Baseline: a903409
Comparison: ee342ae

binary	os	arch	change
agent	linux	amd64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
agent	linux	arm64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
agent	windows	amd64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
agent	darwin	amd64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
agent	darwin	arm64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
iot-agent	linux	amd64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
iot-agent	linux	arm64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
heroku-agent	linux	amd64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
cluster-agent	linux	amd64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent	linux	arm64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent-cloudfoundry	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent-cloudfoundry	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
dogstatsd	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
dogstatsd	linux	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
process-agent	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
process-agent	linux	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
process-agent	windows	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
process-agent	darwin	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
process-agent	darwin	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
heroku-process-agent	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
security-agent	linux	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
security-agent	linux	arm64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
security-agent	windows	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	linux	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	linux	arm64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	windows	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	darwin	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	darwin	arm64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
otel-agent	linux	amd64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
otel-agent	linux	arm64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
privateactionrunner	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
privateactionrunner	linux	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
privateactionrunner	windows	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
privateactionrunner	darwin	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
privateactionrunner	darwin	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers