Skip to content

feat(logs): add pattern extraction to gRPC stateful sender#49219

Draft
DDuongNguyen wants to merge 1 commit into04-10-feat_logs_add_pattern_clustering_tag_management_and_json_processorfrom
04-10-feat_logs_add_pattern_extraction_to_grpc_stateful_sender
Draft

feat(logs): add pattern extraction to gRPC stateful sender#49219
DDuongNguyen wants to merge 1 commit into04-10-feat_logs_add_pattern_clustering_tag_management_and_json_processorfrom
04-10-feat_logs_add_pattern_extraction_to_grpc_stateful_sender

Conversation

@DDuongNguyen
Copy link
Copy Markdown
Contributor

@DDuongNguyen DDuongNguyen commented Apr 10, 2026

What does this PR do?

Integrates pattern extraction into the gRPC stateful sender (built in #48879 by @TheSafo), adding:

  • MessageTranslator with pattern caching (sender.go): Attaches pattern signatures and cluster IDs to outgoing log messages, caching state to avoid redundant computation
  • Batch strategy (batch_strategy.go): Pattern-aware batching that considers signature similarity when grouping messages for transmission
  • Stream worker updates (stream_worker.go): Wires the MessageTranslator into the existing gRPC streaming pipeline
  • Inflight tracking (inflight.go): Tracks in-flight pattern state for acknowledgment handling
  • State telemetry (state_telemetry.go): Telemetry counters for pattern cache hits/misses, encoding decisions
  • Mock helpers (mock_encoder.go, mock_state.go, mock_state_cache_test.go): Test infrastructure for the pattern-aware sender

This is PR 5/6 in a stack. Depends on PR 4 (#49218) for the clustering and tag modules. This PR modifies the gRPC sender from #48879 — it represents only the delta of pattern extraction additions on top of Jake's transport layer.

Motivation

The gRPC sender needs to attach pattern metadata to log messages so the backend can perform stateful encoding (deduplication, compression). This PR adds that translation layer without changing the underlying transport mechanics.

Describe how you validated your changes

  • Unit tests for MessageTranslator caching behavior (mock_state_cache_test.go)
  • Batch strategy tests for pattern-aware grouping
  • Stream worker tests updated for the new encoding flow
  • Metrics module tests for telemetry counters

How to Review this PR

  1. sender.go — the MessageTranslator and its pattern cache
  2. batch_strategy.go — how pattern similarity affects batching
  3. stream_worker.go — diff is small, focus on the MessageTranslator integration points
  4. state_telemetry.go — new telemetry counters
  5. Tests validate each component independently

Additional Notes

This PR modifies files that already exist in #48879's branch. The diff shows only incremental additions (pattern extraction logic), not the full sender implementation.

Copy link
Copy Markdown
Contributor Author

DDuongNguyen commented Apr 10, 2026

@DDuongNguyen DDuongNguyen force-pushed the 04-10-feat_logs_add_pattern_clustering_tag_management_and_json_processor branch from f2d7858 to d0fc212 Compare April 10, 2026 22:06
@DDuongNguyen DDuongNguyen force-pushed the 04-10-feat_logs_add_pattern_extraction_to_grpc_stateful_sender branch from 381377c to aaabc7d Compare April 10, 2026 22:06
@DDuongNguyen DDuongNguyen force-pushed the 04-10-feat_logs_add_pattern_extraction_to_grpc_stateful_sender branch from aaabc7d to f397434 Compare April 14, 2026 16:34
@DDuongNguyen DDuongNguyen force-pushed the 04-10-feat_logs_add_pattern_clustering_tag_management_and_json_processor branch from d0fc212 to 27da999 Compare April 14, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant