feat(logs): add stateful encoding module plumbing and token types

[DDuongNguyen]

What does this PR do?

Lays the foundation for the stateful encoding feature by introducing:

• Token type system (pkg/logs/patterns/token/): Core types — Token, TokenList, Signature, Tokenizer interface — that all downstream pattern modules depend on
• Protobuf definitions (pkg/proto/datadog/stateful/): The StatefulEncoding gRPC service contract and message types for streaming encoded log data
• Configuration (pkg/config/setup/, comp/logs/agent/config/): New logs_config.use_grpc and stateful encoding knobs (max patterns, eviction thresholds, etc.)
• Go module plumbing: go.mod / go.sum replace directives, go.work, and modules.yml updates so the new sub-modules resolve across the monorepo
• DESIGN.md (pkg/logs/patterns/DESIGN.md): Architecture overview of the full feature — read this first for context on the entire stack

This is PR 1/6 in a stack. It is based on #48879 (stateful gRPC transport init by @TheSafo). Review the DESIGN.md for a high-level map of what each subsequent PR adds.

Motivation

The stateful encoding pipeline needs shared types, protobuf contracts, and config wiring before any feature code can be added. Splitting these into the first PR keeps every subsequent PR focused on a single architectural layer.

Describe how you validated your changes

• Existing unit tests for token types pass (pkg/logs/patterns/token/)
• go build ./... succeeds for affected modules
• Proto-generated code compiles and matches the .proto source

How to Review this PR

1. Start with pkg/logs/patterns/DESIGN.md for the full architecture overview
2. Review pkg/proto/datadog/stateful/stateful_encoding.proto for the gRPC contract
3. Skim pkg/logs/patterns/token/ for the core type definitions
4. The go.mod / go.sum changes are mechanical replace directives — spot-check a few

Additional Notes

The bulk of the file count comes from go.mod replace directives (~185 modules). These are additive and mechanical. The actual feature code in this PR is the token package, proto definitions, and config additions.

[DDuongNguyen]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• utf8 #49898 : 3 dependent PRs (#49961 , #50036 , #50047 )
• fix(logs): staging fidelity fixes — tabs, newlines, Unicode positions, UTF-8 safety #49810
• better json #49672
• local proto plus servic #49628
• feat(logs): periodically remove unused entry from dict #49494
• feat(logs): max template size limit + json_as_raw raw bypass #49552
• fix(logs): send delete on tag eviction #49493
• fix(logs): rebuild libpatterns.so targeting glibc 2.30 (was 2.34) #49446
• feat(logs): send JSON via dict-encoded sets of keys #49404
• feat(logs): add dual-send shadow gRPC pipeline #49327
• feat(logs): wire stateful encoding into log agent pipeline #49220
• feat(logs): add pattern extraction to gRPC stateful sender #49219
• feat(logs): add pattern clustering, tag management, and JSON processor #49218
• feat(logs): add generic eviction framework for pattern management #49217
• feat(logs): add Rust FFI tokenizer bridge with FlatBuffers #49216
• feat(logs): add stateful encoding module plumbing and token types #49215 👈 (View in Graphite)
• [Logs Stateful Encoding] transport layer initial implementation #48879
• logs-stateful-main

This stack of pull requests is managed by Graphite. Learn more about stacking.