feat: add opt-in GenAI payload capture (PayloadStore Phase 1)#1981
Open
nickaggarwal wants to merge 1 commit into
Open
feat: add opt-in GenAI payload capture (PayloadStore Phase 1)#1981nickaggarwal wants to merge 1 commit into
nickaggarwal wants to merge 1 commit into
Conversation
Implements Phase 1 of the GenAI payload events proposal (llm-d/llm-d#1206): the PayloadStore interface with the noop and inline backends, opt-in prompt capture on the EPP gateway span, the LLMD_PAYLOAD_* environment surface, and unit tests. When enabled (default off), the gateway records the parsed request as a gen_ai.client.inference.operation.details span event carrying gen_ai.input.messages / gen_ai.system_instructions per the upstream GenAI semantic conventions. Chat-completions, completions and Anthropic-messages shapes are covered; payloads over the inline threshold and blob media parts (no object-store backend until Phase 2) are dropped with llm_d.payload.truncated=true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Nilesh Agarwal <nick.aggarwal@gmail.com>
github-actions
Bot
added
the
size/XXL
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces Phase 1 of an opt-in GenAI request payload capture pipeline for the EPP gateway. It adds a PayloadStore abstraction with noop/inline backends and wires a capturer into the ext_proc request path to emit GenAI semantic-convention span events when explicitly enabled via env vars.
Changes:
- Add
PayloadStorecontract (PayloadRef/PayloadKind) withNoopStoreandInlineStoreimplementations. - Add env-var configuration loading (
LLMD_PAYLOAD_CAPTURE_ENABLED,LLMD_PAYLOAD_BACKEND,LLMD_PAYLOAD_INLINE_THRESHOLD) and aCapturerthat emitsgen_ai.client.inference.operation.detailsevents. - Wire capturer creation into server startup and call capture after scheduling; add unit tests and operator documentation.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/epp/server/runserver.go | Creates a payload capturer from env and injects it into the streaming server when enabled. |
| pkg/epp/payload/store.go | Introduces the PayloadStore interface and Phase 1 noop/inline backends. |
| pkg/epp/payload/store_test.go | Unit tests for NoopStore and InlineStore threshold behavior. |
| pkg/epp/payload/config.go | Env-var config surface and safe fallbacks for payload capture settings. |
| pkg/epp/payload/config_test.go | Tests env-config parsing and capturer gating behavior. |
| pkg/epp/payload/capture.go | Implements extraction + span-event emission for supported request shapes. |
| pkg/epp/payload/capture_test.go | Tests capture behavior (chat/completions/anthropic, multimodal, truncation, gating). |
| pkg/epp/handlers/server.go | Adds capturer field + setter and invokes capture after scheduling. |
| docs/payload-capture.md | Documents configuration, capture semantics, and security considerations. |
Comment on lines
+463
to
+465
| // Opt-in GenAI payload capture: record the prompt on the gateway | ||
| // span once the request has been scheduled. Never fails the request. | ||
| s.payloadCapturer.CaptureRequest(ctx, parseResult.Body) |
Comment on lines
+183
to
+186
| if _, err := c.store.Store(ctx, ref, data); err != nil { | ||
| *truncated = true | ||
| return attribute.KeyValue{}, false | ||
| } |
Comment on lines
+154
to
+158
| if kv, ok := c.inlineJSON(ctx, ref, AttrInputMessages, ext.messages, len(ext.messages) > 0, &ext.truncated); ok { | ||
| attrs = append(attrs, kv) | ||
| } | ||
| if kv, ok := c.inlineJSON(ctx, ref, AttrSystemInstructions, ext.system, len(ext.system) > 0, &ext.truncated); ok { | ||
| attrs = append(attrs, kv) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First implementation PR (Phase 1) of the GenAI payload events proposal (llm-d/llm-d#1206, closes llm-d/llm-d#1108 design): the
PayloadStoreinterface, thenoop/inlinebackends, and opt-in prompt capture on the EPP gateway span — all behind a default-off flag.What's in this PR
PayloadStoreinterface,PayloadRef/PayloadKind,NoopStore,InlineStorepkg/epp/payload/store.goLLMD_PAYLOAD_CAPTURE_ENABLED,LLMD_PAYLOAD_BACKEND,LLMD_PAYLOAD_INLINE_THRESHOLD) with safe fallbackspkg/epp/payload/config.goCapturer: converts the parsed request body togen_ai.input.messages/gen_ai.system_instructionsand records thegen_ai.client.inference.operation.detailsspan eventpkg/epp/payload/capture.goProcess()loop after scheduling, nil-safe and default-offpkg/epp/handlers/server.go,pkg/epp/server/runserver.godocs/payload-capture.mdpkg/epp/payload/*_test.goNote: the proposal sketches the interface at
pkg/telemetry/payload_store.go; it lands here aspkg/epp/payloadto match this repo's layout (EPP-owned per the proposal's ownership section).Behaviour
LLMD_PAYLOAD_CAPTURE_ENABLED=false(or thenoopbackend, the proposal's secondary kill switch) no events are emitted and the request path gains one nil check.backend: inline, the gateway records the prompt on the sampledgateway.requestspan per the upstream GenAI semantic conventions (Opt-In/Developmentattributes). Chat-completions, completions, and Anthropic-messages request shapes are covered.inlineSizeThresholdBytes(default 4096) and blob media parts (data URLs, base64 audio — no object-store backend until Phase 2) are dropped withllm_d.payload.truncated: true; external media URLs are kept as schema-standarduriparts. Capture never fails the request.gcs/s3/filesystembackends are recognised but fall back tonoopwith a startup warning until Phase 2.Testing
go test ./pkg/epp/payload/... ./pkg/epp/handlers/... ./pkg/epp/server/...— passgolangci-lint run(v2.8.0, the CI-pinned version) on the touched packages — 0 issuesgo build ./...— passFollow-ups (per the proposal's phased plan)
GCSStore/S3Store/FilesystemStore+ blob offload with in-placeblob→urirewrite🤖 Generated with Claude Code