feat(logs): add Rust FFI tokenizer bridge with FlatBuffers

[DDuongNguyen]

What does this PR do?

Adds the Rust FFI tokenizer bridge that connects the Go agent to the patterns Rust library via cgo + FlatBuffers:

• Rust tokenizer wrapper (pkg/logs/patterns/tokenizer/rust/): cgo bindings to libpatterns, FlatBuffers schema for zero-copy token serialization, TokenConversion to map Rust tokens → Go token.Token types
• Vendor libraries: Pre-built libpatterns.{dylib,so} for darwin/linux × amd64/arm64, plus the C header
• Build tasks (tasks/patterns.py): Invoke tasks for compiling the Rust library, running benchmarks, and managing vendor artifacts
• Build tag integration (tasks/build_tags.py, tasks/agent.py): patterns build tag gating

This is PR 2/6 in a stack. Depends on PR 1 (#49215) for the token.Token types and Tokenizer interface this bridge implements.

Motivation

The Rust patterns library provides high-performance log tokenization (signature extraction, pattern detection). This bridge lets the Go agent call into it without re-implementing the tokenizer in Go, using FlatBuffers to minimize serialization overhead.

Describe how you validated your changes

• Unit tests for token conversion and roundtrip tokenization (rust_*_test.go)
• FlatBuffers schema compiles and generates correct Go accessors
• tasks/patterns.py successfully builds and vendors the library on macOS arm64

How to Review this PR

1. Start with tokenizer.go — the main RustTokenizer struct and its Tokenize() method
2. Review token_conversion.go for the FlatBuffers → Go token mapping
3. Skim flatbuffers/patterns_tokenizer.fbs for the serialization schema
4. tasks/patterns.py for the build/vendor workflow
5. The vendor/ directory contains pre-built binaries — verify the README for provenance

Additional Notes

The tokenizer_stub.go provides a no-op implementation when the patterns build tag is disabled, so this doesn't affect builds that don't opt in.

[DDuongNguyen]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• utf8 #49898 : 3 dependent PRs (#49961 , #50036 , #50047 )
• fix(logs): staging fidelity fixes — tabs, newlines, Unicode positions, UTF-8 safety #49810
• better json #49672
• local proto plus servic #49628
• feat(logs): periodically remove unused entry from dict #49494
• feat(logs): max template size limit + json_as_raw raw bypass #49552
• fix(logs): send delete on tag eviction #49493
• fix(logs): rebuild libpatterns.so targeting glibc 2.30 (was 2.34) #49446
• feat(logs): send JSON via dict-encoded sets of keys #49404
• feat(logs): add dual-send shadow gRPC pipeline #49327
• feat(logs): wire stateful encoding into log agent pipeline #49220
• feat(logs): add pattern extraction to gRPC stateful sender #49219
• feat(logs): add pattern clustering, tag management, and JSON processor #49218
• feat(logs): add generic eviction framework for pattern management #49217
• feat(logs): add Rust FFI tokenizer bridge with FlatBuffers #49216 👈 (View in Graphite)
• feat(logs): add stateful encoding module plumbing and token types #49215
• [Logs Stateful Encoding] transport layer initial implementation #48879
• logs-stateful-main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[DDuongNguyen]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9957229d8f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Exclude rust_patterns from default Windows agent tags

Adding rust_patterns to the default AGENT_TAGS enables the Rust tokenizer bridge on Windows builds, but this commit does not vendor a Windows libpatterns artifact (the new vendor/README.md marks windows_amd64 as "Not yet built") while tokenizer.go still links -lpatterns from vendor/windows_amd64. In practice, default Windows agent builds can fail at link time with cannot find -lpatterns unless developers manually prepare platform-specific artifacts; this tag should be gated off on Windows (or enabled only when a Windows binary is present).

Useful? React with 👍 / 👎.