From cd382349ac7d20111c505f8ec2b91f4863f8d040 Mon Sep 17 00:00:00 2001 From: Annabell Schaefer Date: Mon, 27 Jul 2026 15:18:26 +0200 Subject: [PATCH 1/3] Add dataset construction skill reference --- .claude-plugin/plugin.json | 2 +- .cursor-plugin/plugin.json | 2 +- skills/langfuse/SKILL.md | 1 + .../references/dataset-construction.md | 43 +++++++++++++++++++ 4 files changed, 46 insertions(+), 2 deletions(-) create mode 100644 skills/langfuse/references/dataset-construction.md diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index bdc19cc..d5726f6 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "langfuse", "description": "Skills for working with Langfuse, the open-source LLM engineering platform for tracing, prompt management, and evaluation.", - "version": "1.3.0", + "version": "1.4.0", "author": { "name": "Langfuse", "email": "support@langfuse.com" diff --git a/.cursor-plugin/plugin.json b/.cursor-plugin/plugin.json index 6b2a88d..e745f7d 100644 --- a/.cursor-plugin/plugin.json +++ b/.cursor-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "langfuse", "displayName": "Langfuse", - "version": "1.3.0", + "version": "1.4.0", "description": "Skills for working with Langfuse — the open-source LLM engineering platform for tracing, prompt management, and evaluation.", "author": { "name": "Langfuse", diff --git a/skills/langfuse/SKILL.md b/skills/langfuse/SKILL.md index a849c11..b8c4102 100644 --- a/skills/langfuse/SKILL.md +++ b/skills/langfuse/SKILL.md @@ -31,6 +31,7 @@ Follow these principles for ALL Langfuse work: ## Use case specific references - instrumenting an existing function/application: references/instrumentation.md +- creating or getting to a good evaluation dataset from production traces, monitored failures, expert examples, existing assets, or targeted synthetic gap-fills: references/dataset-construction.md - migrating prompts from a codebase into Langfuse: references/prompt-migration.md - creating a prompt or changing any part of an existing prompt, including small edits and debugging/tuning: references/prompt-engineering.md - capturing user feedback (thumbs, ratings, implicit signals) as scores on traces: references/user-feedback.md diff --git a/skills/langfuse/references/dataset-construction.md b/skills/langfuse/references/dataset-construction.md new file mode 100644 index 0000000..f559b95 --- /dev/null +++ b/skills/langfuse/references/dataset-construction.md @@ -0,0 +1,43 @@ +--- +name: langfuse-dataset-construction +description: Collaborative Langfuse dataset creation workflow. Use when the user needs to create, design, seed, reshape, review, or upload a Langfuse dataset or dataset version; especially when they need a minimal but complete dataset with agreed input, expected output, metadata, sources, and coverage dimensions before live mutation. +metadata: + required_access: + - CODEBASE + - LANGFUSE_PROJECT_INTERFACE + - LANGFUSE_PROJECT_SCRIPT +--- + +# Langfuse Dataset Construction + +## Primary Guide + +Follow the [Langfuse Academy datasets guide](https://langfuse.com/academy/datasets) as the human-readable source of truth for dataset design. Do not duplicate the guide here; fetch current docs before implementation. + +Use CLI schema/help discovery for current Langfuse API mechanics. Do not hard-code CLI recipes for dataset operations in this reference. + +## Workflow + +Run the dataset workflow step by step: + +1. Read the primary guide and relevant current Langfuse docs. +2. Inspect the codebase, if available, to understand the production path. +3. Inspect the Langfuse project, if available, for existing datasets, traces, prompts, scores, monitors, and user feedback. +4. Clarify the user's goal, using the guide as the interview frame. +5. Clarify available sources: production traces, existing assets, expert examples, user feedback, tickets, synthetic gap-fills, or benchmark data. +6. Propose one or more dataset directions with tradeoffs. +7. Let the user choose the direction before live mutation. +8. Confirm runtime, environment, credentials availability, and cost/rate-limit expectations before running SDK code. +9. Build the smallest approved draft, run or upload it, audit the result, then review and repeat. + +## Rules + +- Do not create, upsert, reshape, or upload a live Langfuse dataset until the user has approved the dataset goal, source mix, item schema, and first minimal draft, unless the user already gave those details and explicitly asked for immediate mutation. +- Design the smallest complete dataset version that can serve the goal. Prefer a minimal reviewable v0 over broad coverage. A first draft is usually 5-12 high-signal items unless the user asks for a different size. +- Inspect real code and traces when available to understand the production path. Explain whether the proposed dataset input is literal production-shaped input or a normalized experiment input, and why. +- Agree on the input shape before writing or uploading items. The input must be readable, stable, and directly mappable into the experiment runner or application path. +- Agree on the expected output shape before writing or uploading items. The expected output must support the planned evaluator or review rubric. +- Agree on the metadata shape before writing or uploading items. Include at least source, category or dimension labels, difficulty when useful, dataset_role, provenance, and review status when applicable. +- Propose input distribution dimensions and ask the user to edit them. Common dimensions: intent, topic, source, difficulty, language, context availability, edge case, failure mode, risk level, and dataset_role. +- Ask the user to confirm the goal, input shape, expected output shape, metadata fields, distribution dimensions, and source mix before upload. +- After approved upload, verify dataset name, version or schema metadata, item count, schema presence, source distribution, and a sample item. Report only verified state. From cae3fe4ac1510a98791c10c095a6ff92f6131715 Mon Sep 17 00:00:00 2001 From: Annabell Schaefer Date: Thu, 30 Jul 2026 16:15:28 +0200 Subject: [PATCH 2/3] Refine Langfuse skill references --- skills/langfuse/SKILL.md | 1 + .../references/dataset-construction.md | 27 +++++------- .../langfuse/references/improvement-loop.md | 41 +++++++++++++++++++ 3 files changed, 52 insertions(+), 17 deletions(-) create mode 100644 skills/langfuse/references/improvement-loop.md diff --git a/skills/langfuse/SKILL.md b/skills/langfuse/SKILL.md index b8c4102..2aa1fee 100644 --- a/skills/langfuse/SKILL.md +++ b/skills/langfuse/SKILL.md @@ -40,6 +40,7 @@ Follow these principles for ALL Langfuse work: - judge calibration (LLM-as-a-Judge reliability, simple accuracy checks, advanced split-based validation, confusion matrices, and metric ingestion): references/judge-calibration.md - systematic error analysis — reading traces, building failure taxonomy, deciding what to fix: references/error-analysis.md - issue detection triage — find production problems and rank them P0–P3: references/issue-detection-triage.md +- closing the loop from a reported symptom to a measured, proven fix: references/improvement-loop.md - setting up CI/CD experiment gates with `langfuse/experiment-action`: references/ci-cd.md - submitting feedback about this skill: references/skill-feedback.md diff --git a/skills/langfuse/references/dataset-construction.md b/skills/langfuse/references/dataset-construction.md index f559b95..406d5a1 100644 --- a/skills/langfuse/references/dataset-construction.md +++ b/skills/langfuse/references/dataset-construction.md @@ -3,7 +3,6 @@ name: langfuse-dataset-construction description: Collaborative Langfuse dataset creation workflow. Use when the user needs to create, design, seed, reshape, review, or upload a Langfuse dataset or dataset version; especially when they need a minimal but complete dataset with agreed input, expected output, metadata, sources, and coverage dimensions before live mutation. metadata: required_access: - - CODEBASE - LANGFUSE_PROJECT_INTERFACE - LANGFUSE_PROJECT_SCRIPT --- @@ -18,26 +17,20 @@ Use CLI schema/help discovery for current Langfuse API mechanics. Do not hard-co ## Workflow -Run the dataset workflow step by step: +Guide the user through dataset creation as an interview, proposal, approval, implementation loop: 1. Read the primary guide and relevant current Langfuse docs. -2. Inspect the codebase, if available, to understand the production path. -3. Inspect the Langfuse project, if available, for existing datasets, traces, prompts, scores, monitors, and user feedback. -4. Clarify the user's goal, using the guide as the interview frame. -5. Clarify available sources: production traces, existing assets, expert examples, user feedback, tickets, synthetic gap-fills, or benchmark data. -6. Propose one or more dataset directions with tradeoffs. -7. Let the user choose the direction before live mutation. -8. Confirm runtime, environment, credentials availability, and cost/rate-limit expectations before running SDK code. -9. Build the smallest approved draft, run or upload it, audit the result, then review and repeat. +2. Inspect available context: the user's goal, the application path or codebase when available, existing datasets, traces, prompts, scores, monitors, user feedback, tickets, expert examples, existing eval assets, and synthetic gaps. +3. Clarify the release or evaluation decision the dataset should support. +4. Propose the source mix, input distribution, input shape, expected output shape, metadata fields, evaluator or run plan, and minimal v0 size. +5. Get the user's approval before creating, reshaping, uploading, or running live project data. +6. Build the smallest approved draft, run or upload it, verify readback, review results with the user, and iterate. -## Rules +## Critical Rules +- Interview the user on available source data beyond traces. +- Interview the user about the goal they are trying to achieve and the problem they face. - Do not create, upsert, reshape, or upload a live Langfuse dataset until the user has approved the dataset goal, source mix, item schema, and first minimal draft, unless the user already gave those details and explicitly asked for immediate mutation. - Design the smallest complete dataset version that can serve the goal. Prefer a minimal reviewable v0 over broad coverage. A first draft is usually 5-12 high-signal items unless the user asks for a different size. - Inspect real code and traces when available to understand the production path. Explain whether the proposed dataset input is literal production-shaped input or a normalized experiment input, and why. -- Agree on the input shape before writing or uploading items. The input must be readable, stable, and directly mappable into the experiment runner or application path. -- Agree on the expected output shape before writing or uploading items. The expected output must support the planned evaluator or review rubric. -- Agree on the metadata shape before writing or uploading items. Include at least source, category or dimension labels, difficulty when useful, dataset_role, provenance, and review status when applicable. -- Propose input distribution dimensions and ask the user to edit them. Common dimensions: intent, topic, source, difficulty, language, context availability, edge case, failure mode, risk level, and dataset_role. -- Ask the user to confirm the goal, input shape, expected output shape, metadata fields, distribution dimensions, and source mix before upload. -- After approved upload, verify dataset name, version or schema metadata, item count, schema presence, source distribution, and a sample item. Report only verified state. +- Keep `input`, `expectedOutput`, and `metadata` responsibilities separate. Put additional information, notes, and comments into `metadata`, not into `input` or `expectedOutput`. diff --git a/skills/langfuse/references/improvement-loop.md b/skills/langfuse/references/improvement-loop.md new file mode 100644 index 0000000..9046f3e --- /dev/null +++ b/skills/langfuse/references/improvement-loop.md @@ -0,0 +1,41 @@ +--- +name: langfuse-improvement-loop +description: Symptom-to-fix Langfuse workflow. Use when the user reports an LLM or agent application problem and wants to identify the root cause, choose the right fix, encode failures as evaluation coverage, and prove improvement with dataset runs before promotion. +metadata: + required_access: + - CODEBASE + - LANGFUSE_PROJECT_INTERFACE + - LANGFUSE_PROJECT_SCRIPT +--- + +# Langfuse Improvement Loop + +## Current Sources + +Fetch current Langfuse docs before implementation. Use the docs for tracing, scores, datasets, dataset runs, evaluations, and prompt management when the candidate fix touches prompts. + +Use CLI schema/help discovery for current Langfuse API mechanics. Do not hard-code CLI recipes here. + +## Workflow + +Run the improvement loop step by step: + +1. Restate the reported symptom as a measurable trace, score, evaluator, or user-feedback pattern over a defined time window. +2. Inspect Langfuse project data to find concrete examples and confirm whether the symptom repeats. +3. Identify the likely root cause from traces and, when available, source code or prompt behavior. +4. Propose fix options with tradeoffs: prompt, retrieval, context assembly, tool wiring, application logic, model choice, or measurement first. +5. Get the user's approval for the selected fix path and any live project mutation. +6. Make the candidate change behind a reversible boundary such as a branch, feature flag, non-production prompt label, or isolated runner setting. +7. Encode the failing cases as dataset items or a targeted evaluator, reusing the dataset-construction reference when dataset design is involved. +8. Compare baseline and candidate runs on the same dataset. Inspect per-item regressions, not only aggregate scores. +9. Report verified evidence, residual risk, and whether the fix should ship. Ask before promoting anything to production. + +## Critical Rules + +- Start from real examples; do not jump from a vague symptom to a fix. +- Keep diagnosis, implementation, and measurement separate enough that the result is interpretable. +- Prefer the smallest reversible fix that addresses the repeated pattern. +- Do not publish prompts to production, mutate production configuration, or change live routing without explicit user approval. +- If the project lacks a meaningful dataset, evaluator, or score for the symptom, create the measurement plan before recommending a product change. +- Treat production traces as evidence, not ground truth. Review sensitive data handling before adding traces to datasets. +- Verify any dataset, prompt, score, or run changes by reading them back from Langfuse before reporting them as done. From 9734f5ca0f4a012f1160d539086d961288df93ad Mon Sep 17 00:00:00 2001 From: Annabell Schaefer Date: Fri, 31 Jul 2026 17:06:29 +0200 Subject: [PATCH 3/3] Clean up issue triage gotchas heading --- skills/langfuse/references/issue-detection-triage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/langfuse/references/issue-detection-triage.md b/skills/langfuse/references/issue-detection-triage.md index 81af7f8..de13b28 100644 --- a/skills/langfuse/references/issue-detection-triage.md +++ b/skills/langfuse/references/issue-detection-triage.md @@ -205,7 +205,7 @@ So two runs over the same data report the same numbers, define metrics consisten - **Error rate** — affected items / items examined, with the denominator stated (traces, or calls of a specific tool). Count failures from output **content**, not just `level=ERROR` (see #1 in Reliability). - **Prevalence** — "traces affected / traces examined" against the *product* traces you inspected, not the raw window count. If synthetic traffic dominates (workflow step 3), report the rate both including and excluding it when it changes the conclusion. -### Gotchas (learned the hard way) +### Gotchas - **Pagination differs per endpoint** (`traces`/`scores` are page-based, `observations` is cursor-based) — follow the pagination tip in [cli.md](cli.md) exactly. - **Trace `scores` field returns score IDs (strings), not score objects.** To get values, pull scores separately and join on `traceId`.