feat: route skill feedback through Langfuse intake #76
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,55 +1,35 @@ | ||
| --- | ||
| name: langfuse-skill-feedback | ||
| description: Submit feedback about the Langfuse skill to its maintainers. Use when the skill's instructions are wrong, outdated, missing something, or could be improved. | ||
| metadata: | ||
| required_access: | ||
| - LANGFUSE_PROJECT_INTERFACE | ||
| --- | ||
|
|
||
| # Skill Feedback | ||
|
|
||
| Use this only for feedback about the skill itself, not Langfuse the product or a user's application data. | ||
|
|
||
| 1. Draft concise feedback using the following structure: | ||
| - **Describe your idea or feedback** (required) | ||
| A clear description of what went wrong or what could be improved. Include: | ||
| - What the user was trying to do (as `goal`) | ||
| - What the skill did vs what was expected | ||
| - Any specific instructions that were incorrect or missing | ||
|
|
||
| - **What would the ideal outcome look like?** (optional) | ||
| What the correct behavior or guidance should be. | ||
|
|
||
| If feedback targets existing skill, reference in `target`. | ||
| 2. If the user wants a reply, ask them to include an email address in `feedback`; use only an address they explicitly provide. | ||
| 3. Show every submitted field exactly as it will be sent and ask for explicit permission. Do not submit without approval. | ||
|
Check warning on line 25 in skills/langfuse/references/skill-feedback.md
|
||
|
|
||
| ## Submission options | ||
|
|
||
| Use the first available option, or the option the user prefers: | ||
|
|
||
| 1. **Authenticated Langfuse MCP, CLI, or Public API** — prefer the `submitFeedback` tool on the Langfuse MCP server. If it is unavailable, discover the current feedback operation with the Langfuse CLI schema/help and submit through the authenticated Public API. Do not ask users to paste credentials into chat. | ||
| 2. **Langfuse Docs MCP** — use its unauthenticated `submitFeedback` tool when no authenticated Langfuse interface is available. | ||
| 3. **GitHub issue or discussion** — if no MCP/CLI/API is available, or the user wants a public, trackable thread, provide a prefilled discussion link for the user to submit: `https://github.com/langfuse/skills/discussions/new?category=ideas-improvements&title=<url-encoded title>&body=<url-encoded body>`. Use `https://github.com/langfuse/skills/issues/new` if they prefer an issue. | ||
|
Check warning on line 33 in skills/langfuse/references/skill-feedback.md
|
||
|
There was a problem hiding this comment. 🟡 The prefilled GitHub discussion fallback link includes Extended reasoning...The bugLine 33 of Why this is inconsistent, not intentionalGitHub's Why the surrounding flow doesn't prevent the problemThe whole point of this reference's draft-then-approve flow (steps 1-3) is that the user signs off on exact wording once, and every submission path downstream should just deliver that approved content. Two of the three "Submission options" bullets do this correctly (the authenticated MCP/CLI path sends the drafted fields directly; the discussion link prefills them via query params). The issue link silently breaks that guarantee for anyone who happens to prefer issues over discussions, or for repos/orgs where discussions are disabled and issue is the only GitHub option. Step-by-step proof
FixAppend the same encoded params to the issue link, mirroring the discussion link: On severityThis is instruction-only content, nothing crashes, and it only affects the secondary "if they prefer an issue" branch of a fallback path that itself only triggers when no authenticated Langfuse interface is available. A capable agent could plausibly infer by analogy that "prefilled" should apply to both links. That keeps this at nit rather than blocking-severity — worst case is a minor retype inconvenience, not a broken workflow. |
||
|
|
||
| Report the Langfuse correlation ID or GitHub URL. If submission fails, give a safe error without exposing credentials or request bodies. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 The explicit instruction to strip secrets, credentials, and trace payloads before drafting feedback (present in an earlier version of this file) was dropped during the restructure, even though the PR description promises the workflow will 'prohibit secrets, customer data, trace payloads, and unrelated context.' The remaining scoping line and the mandatory show-and-approve step (step 3) mitigate this but rely on the user catching sensitive content themselves rather than the agent proactively stripping it before drafting — worth restoring a one-line reminder, especially since option 3 can post the draft publicly to GitHub.
Extended reasoning...
An earlier draft of this file (git history, commit 7634a5f) contained an explicit step: "Remove secrets, credentials, customer data, trace payloads, and unrelated context." Commit b5b4533 restructured the drafting steps and this sentence was dropped; it is absent from the final version at lines 13-25, which now cover only draft structure (step 1), the optional reply email (step 2), and the show-and-approve permission gate (step 3). Meanwhile the PR description's own summary still promises the workflow will "prohibit secrets, customer data, trace payloads, and unrelated context" — so the shipped instructions no longer match the stated intent of the PR, most plausibly as an accidental casualty of the rewrite rather than a deliberate decision to relax the guardrail.
This matters because Submission options option 3 posts the drafted feedback body publicly as a GitHub discussion or issue, and option 1 sends it to the Langfuse backend. An agent drafting feedback from conversation context — e.g. quoting a failing CLI invocation, an error message, or a trace snippet to explain "what the skill did vs what was expected" — has no explicit instruction telling it to scrub that content for embedded secrets or credentials before it ever reaches the draft.
Concretely walking through it: (1) a user hits a bug where a skill instructed them to run a CLI command against an API endpoint, and that command (as run) included an auth token in a flag or environment reference; (2) the user pastes the failing command into chat while describing the problem; (3) the agent, per step 1, quotes "what the skill did vs what was expected" and includes the command verbatim in the draft to make the report concrete; (4) step 3 shows the drafted fields to the user and asks for permission — but the instruction is only to display the field, not to actively check it for secrets, so a user skimming a familiar-looking command may approve without noticing the embedded token; (5) if option 3 is used, that token is now published publicly on GitHub.
The remaining guardrails do meaningfully reduce, but don't close, this gap. Line 11 ("not... a user's application data") scopes out trace payloads and customer data at a topical level, but doesn't clearly cover a credential incidentally embedded in an otherwise on-topic report about the skill itself. Line 31 ("do not ask users to paste credentials into chat") and line 35 (safe error messages) guard against the agent soliciting credentials or leaking them on failure, not against the agent including something the user already pasted. Step 3's show-and-approve gate keeps a human in the loop, but it is a passive review step, not an instruction for the agent to actively strip sensitive content the way the deleted sentence did — it shifts the entire burden onto the user noticing at review time.
The fix is cheap: restore a short instruction in step 1 (or as a standalone step) telling the agent to strip secrets, credentials, and trace/application-data snippets from the drafted text before presenting it for approval, matching what the PR description already promises.
Given that step 3's mandatory human approval gate does stand between drafting and any submission, and nothing crashes or silently leaks data without a user clicking approve, this is best treated as a content regression rather than a merge-blocking defect.