Add dataset construction skill reference

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "langfuse",
   "description": "Skills for working with Langfuse, the open-source LLM engineering platform for tracing, prompt management, and evaluation.",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "author": {
     "name": "Langfuse",
     "email": "support@langfuse.com"

.cursor-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "langfuse",
   "displayName": "Langfuse",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "description": "Skills for working with Langfuse — the open-source LLM engineering platform for tracing, prompt management, and evaluation.",
   "author": {
     "name": "Langfuse",

skills/langfuse/SKILL.md

@@ -31,6 +31,7 @@ Follow these principles for ALL Langfuse work:
 ## Use case specific references
 
 - instrumenting an existing function/application: references/instrumentation.md
+- creating or getting to a good evaluation dataset from production traces, monitored failures, expert examples, existing assets, or targeted synthetic gap-fills: references/dataset-construction.md
 - migrating prompts from a codebase into Langfuse: references/prompt-migration.md
 - creating a prompt or changing any part of an existing prompt, including small edits and debugging/tuning: references/prompt-engineering.md
 - capturing user feedback (thumbs, ratings, implicit signals) as scores on traces: references/user-feedback.md
@@ -39,6 +40,7 @@ Follow these principles for ALL Langfuse work:
 - judge calibration (LLM-as-a-Judge reliability, simple accuracy checks, advanced split-based validation, confusion matrices, and metric ingestion): references/judge-calibration.md
 - systematic error analysis — reading traces, building failure taxonomy, deciding what to fix: references/error-analysis.md
 - issue detection triage — find production problems and rank them P0–P3: references/issue-detection-triage.md
+- closing the loop from a reported symptom to a measured, proven fix: references/improvement-loop.md
 - setting up CI/CD experiment gates with `langfuse/experiment-action`: references/ci-cd.md
 - submitting feedback about this skill: references/skill-feedback.md
 

skills/langfuse/references/dataset-construction.md

@@ -0,0 +1,36 @@
+---
+name: langfuse-dataset-construction
+description: Collaborative Langfuse dataset creation workflow. Use when the user needs to create, design, seed, reshape, review, or upload a Langfuse dataset or dataset version; especially when they need a minimal but complete dataset with agreed input, expected output, metadata, sources, and coverage dimensions before live mutation.
+metadata:
+  required_access:
+    - LANGFUSE_PROJECT_INTERFACE
+    - LANGFUSE_PROJECT_SCRIPT
+---
+
+# Langfuse Dataset Construction
+
+## Primary Guide
+
+Follow the [Langfuse Academy datasets guide](https://langfuse.com/academy/datasets) as the human-readable source of truth for dataset design. Do not duplicate the guide here; fetch current docs before implementation.
+
+Use CLI schema/help discovery for current Langfuse API mechanics. Do not hard-code CLI recipes for dataset operations in this reference.
+
+## Workflow
+
+Guide the user through dataset creation as an interview, proposal, approval, implementation loop:
+
+1. Read the primary guide and relevant current Langfuse docs.
+2. Inspect available context: the user's goal, the application path or codebase when available, existing datasets, traces, prompts, scores, monitors, user feedback, tickets, expert examples, existing eval assets, and synthetic gaps.
+3. Clarify the release or evaluation decision the dataset should support.
+4. Propose the source mix, input distribution, input shape, expected output shape, metadata fields, evaluator or run plan, and minimal v0 size.
+5. Get the user's approval before creating, reshaping, uploading, or running live project data.
+6. Build the smallest approved draft, run or upload it, verify readback, review results with the user, and iterate.
+
+## Critical Rules
+
+- Interview the user on available source data beyond traces.
+- Interview the user about the goal they are trying to achieve and the problem they face.
+- Do not create, upsert, reshape, or upload a live Langfuse dataset until the user has approved the dataset goal, source mix, item schema, and first minimal draft, unless the user already gave those details and explicitly asked for immediate mutation.
+- Design the smallest complete dataset version that can serve the goal. Prefer a minimal reviewable v0 over broad coverage. A first draft is usually 5-12 high-signal items unless the user asks for a different size.
+- Inspect real code and traces when available to understand the production path. Explain whether the proposed dataset input is literal production-shaped input or a normalized experiment input, and why.
+- Keep `input`, `expectedOutput`, and `metadata` responsibilities separate. Put additional information, notes, and comments into `metadata`, not into `input` or `expectedOutput`.

skills/langfuse/references/improvement-loop.md

@@ -0,0 +1,41 @@
+---
+name: langfuse-improvement-loop
+description: Symptom-to-fix Langfuse workflow. Use when the user reports an LLM or agent application problem and wants to identify the root cause, choose the right fix, encode failures as evaluation coverage, and prove improvement with dataset runs before promotion.
+metadata:
+  required_access:
+    - CODEBASE
+    - LANGFUSE_PROJECT_INTERFACE
+    - LANGFUSE_PROJECT_SCRIPT
+---
+
+# Langfuse Improvement Loop
+
+## Current Sources
+
+Fetch current Langfuse docs before implementation. Use the docs for tracing, scores, datasets, dataset runs, evaluations, and prompt management when the candidate fix touches prompts.
+
+Use CLI schema/help discovery for current Langfuse API mechanics. Do not hard-code CLI recipes here.
+
+## Workflow
+
+Run the improvement loop step by step:
+
+1. Restate the reported symptom as a measurable trace, score, evaluator, or user-feedback pattern over a defined time window.
+2. Inspect Langfuse project data to find concrete examples and confirm whether the symptom repeats.
+3. Identify the likely root cause from traces and, when available, source code or prompt behavior.
+4. Propose fix options with tradeoffs: prompt, retrieval, context assembly, tool wiring, application logic, model choice, or measurement first.
+5. Get the user's approval for the selected fix path and any live project mutation.
+6. Make the candidate change behind a reversible boundary such as a branch, feature flag, non-production prompt label, or isolated runner setting.
+7. Encode the failing cases as dataset items or a targeted evaluator, reusing the dataset-construction reference when dataset design is involved.
+8. Compare baseline and candidate runs on the same dataset. Inspect per-item regressions, not only aggregate scores.
+9. Report verified evidence, residual risk, and whether the fix should ship. Ask before promoting anything to production.
+
+## Critical Rules
+
+- Start from real examples; do not jump from a vague symptom to a fix.
+- Keep diagnosis, implementation, and measurement separate enough that the result is interpretable.
+- Prefer the smallest reversible fix that addresses the repeated pattern.
+- Do not publish prompts to production, mutate production configuration, or change live routing without explicit user approval.
+- If the project lacks a meaningful dataset, evaluator, or score for the symptom, create the measurement plan before recommending a product change.
+- Treat production traces as evidence, not ground truth. Review sensitive data handling before adding traces to datasets.
+- Verify any dataset, prompt, score, or run changes by reading them back from Langfuse before reporting them as done.

skills/langfuse/references/issue-detection-triage.md

@@ -205,7 +205,7 @@ So two runs over the same data report the same numbers, define metrics consisten
 - **Error rate** — affected items / items examined, with the denominator stated (traces, or calls of a specific tool). Count failures from output **content**, not just `level=ERROR` (see #1 in Reliability).
 - **Prevalence** — "traces affected / traces examined" against the *product* traces you inspected, not the raw window count. If synthetic traffic dominates (workflow step 3), report the rate both including and excluding it when it changes the conclusion.
 
-### Gotchas (learned the hard way)
+### Gotchas
 
 - **Pagination differs per endpoint** (`traces`/`scores` are page-based, `observations` is cursor-based) — follow the pagination tip in [cli.md](cli.md) exactly.
 - **Trace `scores` field returns score IDs (strings), not score objects.** To get values, pull scores separately and join on `traceId`.