From 84cd0c0418d25d3a52244ff7dfae75d9bd2ccf71 Mon Sep 17 00:00:00 2001 From: Rudra Dudhat Date: Fri, 19 Jun 2026 22:42:35 +0530 Subject: [PATCH 1/2] docs: fold validated product direction into v2 spec Adds Agent debugger module (sec 6.4), rewrites sec 7 into the six product surfaces + two-ways-in entry + evidence-over-abstraction principle, resolves the frontend decision (Vite/React, warm-editorial matching the portfolio), adds agent/trace CLI commands, and gitignores .superpowers/ + .diffprompt_cache/. Co-Authored-By: Claude Opus 4.8 --- .gitignore | 2 + ...2026-06-19-diffprompt-v2-program-design.md | 65 +++++++++++++++---- 2 files changed, 55 insertions(+), 12 deletions(-) diff --git a/.gitignore b/.gitignore index 6d03986..49ac1e1 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,5 @@ build/ venv/ diffprompt_report.* diffprompt.ontology.json +.diffprompt_cache/ +.superpowers/ diff --git a/docs/superpowers/specs/2026-06-19-diffprompt-v2-program-design.md b/docs/superpowers/specs/2026-06-19-diffprompt-v2-program-design.md index 49659a8..1ddbbb8 100644 --- a/docs/superpowers/specs/2026-06-19-diffprompt-v2-program-design.md +++ b/docs/superpowers/specs/2026-06-19-diffprompt-v2-program-design.md @@ -133,10 +133,39 @@ diffprompt/ - **Output**: provenance trace per agent action; web view of the influence breakdown. - **Risks**: influence without attention access is approximate (counterfactual is the honest method; state limits clearly); agent-framework coupling (keep core agnostic). -## 7. Web UI / service -- **Backend**: FastAPI (existing user expertise). REST for reports/baselines/traces; SSE/WebSocket for live runs and drift updates. Single-tenant/self-host in v2. -- **Frontend**: **OPEN DECISION (see §10)** — proposed Vite + React + Tailwind, aesthetic matched to the portfolio theme (verify: memory conflicts cinematic-gold vs beige-editorial). Views: diff report viewer, attribution explorer, drift timeline, provenance trace. -- **Packaging**: `diffprompt serve` boots API + static dashboard; Docker image for self-host. +### 6.4 Agent debugger (new pillar) — "where did the agent go wrong?" +Emerged during product design; the agentic-reliability flagship and the strongest tie to the niche (tool poisoning, memory/injection, multi-agent chains). +- **Trajectory capture**: a multi-step agent run (plan → tool call → reasoning → action → response) as ordered, typed steps, each with status (ok/warn/fail), tokens, latency. +- **Root-cause step**: pinpoint the step where the trajectory derailed (e.g. treated a tool result as an instruction), not just that the final output was wrong. +- **Built on the other modules** — step-level context reuses **Provenance** (per-source influence + trust); the bad outcome is scored with the diff/judge; **replay-from-step-N with a fix applied** (quarantine a source, edit the prompt) reuses the **counterfactual ablation**. +- **Trajectory diff**: compare two agent runs on the same task (v1 vs v2 agent), see where they diverged. +- **Entry**: instrument once (`@trace` decorator / OTLP exporter / LangGraph callback) or upload a `trace.json`. +- **Output**: trajectory timeline, root-cause pin, replay-from-step, "add as a regression test". +- **Risks**: framework coupling (keep the core trajectory model agnostic, adapters per framework); replay fidelity (deterministic tool stubs for replayed steps). + +## 7. Product surfaces & UX + +**Design principle — evidence over abstraction.** Every screen ends in evidence or an action: see the behavior → find the cause → apply a fix → re-verify. Abstract dashboards (radar/bubble charts) are demoted to a secondary "profile" view; the spine is the actual outputs, the actual prompt sentence, the actual trajectory step. (This was an explicit course-correction during design — early chart-only mockups were rejected as "too abstract.") + +**Entry — two ways in** (`diffprompt` composer, mirrors the CLI): +- **Prompt** — paste v1/v2 (or load from a file / GitHub PR), choose provider · model · judge · test source (auto-generate N *or* upload real `.jsonl`), Run. → feeds Diff / Fix / Drift. +- **Agent** — not pasted; instrument once (`@trace` / OTLP / LangGraph callback) or drop a `trace.json`. → feeds Agent / Provenance / Trace. + +**Six surfaces, one engine** (they share the trajectory model, context-provenance, and counterfactual replay): + +| Surface | Job | Signature element | +|---|---|---| +| **Diff** | did v2 help or hurt? | verdict-first banner + behavioral slices | +| **Fix** | exactly what to change | annotated prompt (top-4 by \|impact\|, rest collapsed) + word-level output diff + auto-generated v3 + projected score + one-click fix-and-re-test | +| **Drift** | did the model change under me? | same-probe baseline-vs-today output diff + concrete shift metrics + alerting | +| **Trace** | where did time/money go; what failed quietly | OTel run waterfall + cost/tokens/reliability + Langfuse/Phoenix export | +| **Agent** | where did the agent go wrong | trajectory timeline + root-cause step + replay-from-step | +| **Provenance** | why did it do that / was it hijacked | per-source influence + trust + verbatim injected text + counterfactual | + +**Design system (RESOLVED — matches the portfolio, warm editorial; not the earlier dark/gold).** Cream paper `#f3ecdf`, rust/terracotta accent `#b5532f`, high-contrast serif display (Fraunces) with rust-italic emphasis, JetBrains Mono for data/code, black pill buttons, label/value cards with thin dividers, a dark mono ticker with `✦` separators, subtle outlined deco circles. Reference mockups (entry + all six surfaces) live in `.superpowers/brainstorm/`. + +**Frontend stack:** Vite + React + Tailwind (lightweight, no SSR needed for a self-host dashboard). +**Backend & packaging:** FastAPI; REST for reports/baselines/traces, SSE/WebSocket for live runs + drift. `diffprompt serve` boots API + static dashboard; Docker image for self-host. Single-tenant in v2. ## 8. Data model (store) Entities: `Run`, `Report`, `Baseline`, `Fingerprint`, `Attribution`, `ProvenanceTrace`, `Alert`. SQLite default (single file, zero-config); Postgres for the service. Pydantic models already define the report shape — extend, version with a schema migration tool (Alembic for Postgres path). @@ -149,22 +178,24 @@ diffprompt baseline create

# Idea 2 diffprompt drift check diffprompt watch diffprompt provenance # Idea 3 -diffprompt serve # web + API +diffprompt agent # agent debugger (trajectory + root cause + replay) +diffprompt trace # observability waterfall for a run +diffprompt serve # web + API (the composer + six surfaces) diffprompt replay # cache replay ``` Consistent flags across commands: `--model`, `--judge`, `--provider`, `--local-only`, `--output`, `-v/-vv`. -## 10. Open decisions (need your call) -1. **Frontend stack & theme** — Vite+React+Tailwind vs Next.js; and which portfolio aesthetic to match (cinematic-gold vs beige-editorial — memory conflicts, must verify against the live site). -2. **Judge default** — keep Groq 8b fallback, or require a stronger judge by default given observed over-leniency? Affects accuracy goal. -3. **Self-host only vs hosted later** — confirm v2 is single-tenant/self-host (no auth/billing). +## 10. Open decisions +1. ~~Frontend stack & theme~~ — **RESOLVED**: Vite + React + Tailwind; warm-editorial design system matching the portfolio (see §7). Mockups validated for entry + all six surfaces. +2. **Judge default** *(still open)* — keep Groq 8b fallback, or require a stronger judge by default given observed over-leniency (live run scored 100/100)? Affects the accuracy goal. The Phase-0 judge eval harness (§5.7) will quantify this. +3. **Self-host only vs hosted later** *(still open)* — confirm v2 is single-tenant/self-host (no auth/billing). ## 11. Phasing (build order; all speced deep, sequenced for delivery) - **Phase 0 — Foundation**: providers, retry, logging, streaming orchestrator, telemetry, caching, test infra + CI gates. *Everything else depends on this.* - **Phase 1 — Attribution**: pure analysis on the new engine; fastest standalone win after foundation. - **Phase 2 — Drift**: store + scheduler + worker pool + alerting + first web views (the observability hero). -- **Phase 3 — Provenance**: source tagging + ablation + trust model + LangGraph adapter. -- **Phase 4 — Web maturation**: unify all modules in the dashboard, Docker self-host, polish. +- **Phase 3 — Provenance + Agent debugger**: source tagging + ablation + trust model + LangGraph adapter; trajectory capture, root-cause detection, replay-from-step. These two share the counterfactual machinery, so they ship together. +- **Phase 4 — Web maturation**: the composer entry + unify all six surfaces in the dashboard, Docker self-host, polish. ## 12. Success metrics - Speed: n=20 diff < 30s warm; first event < 3s. @@ -174,7 +205,7 @@ Consistent flags across commands: `--model`, `--judge`, `--provider`, `--local-o - Traction: stars, and at least one external user running drift monitoring. ## 13. Risks (program-level) -- **Scope** — four phases is large; Phase 0 must stay disciplined or everything slips. Mitigate: ship each phase behind its own milestone, each independently useful. +- **Scope** — the plan is large (5 phases, 6 surfaces); Phase 0 must stay disciplined or everything slips. Mitigate: ship each phase behind its own milestone, each independently useful. - **Cost/limits** — Shapley + drift probes multiply LLM calls. Mitigate: cache, budgets, rate limiter, local-first. - **Judge quality** — the accuracy goal hinges on it; the eval harness is a Phase-0 gate, not an afterthought. - **Web maintenance** — a dashboard is ongoing surface area; keep it thin and generated from the same event model. @@ -183,3 +214,13 @@ Consistent flags across commands: `--model`, `--judge`, `--provider`, `--local-o ## Appendix A — mapping to `future-ideas/IDEAS.md` - Idea 1 → §6.1 Attribution. Idea 2 → §6.2 Drift. Idea 3 → §6.3 Provenance. The repo's own notes (Idea 2 fastest to users, Idea 1 deepest interview story, Idea 3 most unique) are reflected in the phasing rationale. +- §6.4 **Agent debugger** is a new pillar added during product design — not in the original IDEAS.md. It composes Provenance + the counterfactual into a trajectory debugger and is the clearest single expression of the agentic-reliability niche. + +## Appendix B — validated mockups +Interactive mockups built and reviewed via the brainstorming visual companion (warm-editorial), persisted in `.superpowers/brainstorm/`: +- `composer-entry.html` — the two-ways-in entry (prompt paste vs agent instrument) +- `fix-the-prompt.html` — annotated prompt + word-level output diff + auto-v3 +- `drift-monitor.html` — same-probe baseline-vs-today + shift metrics +- `observability-trace.html` — OTel run waterfall + cost/reliability + Langfuse export +- `agent-debugger.html` — trajectory timeline + root-cause + replay +- `provenance-trace.html` — per-source influence + injected text + counterfactual From e1ab926e8e2c9880edb007b08da06a0dfd51c3d5 Mon Sep 17 00:00:00 2001 From: Rudra Dudhat Date: Fri, 19 Jun 2026 22:47:45 +0530 Subject: [PATCH 2/2] docs: add Red-team / security module (sec 6.5) to v2 spec Auto jailbreak/injection testing against a system prompt (OWASP LLM Top 10 + garak families), shows the verbatim attacks that succeeded + what leaked, auto-hardens and re-tests, with a CI gate on CRITICALs. Now seven surfaces. Updates the surfaces table, CLI grammar, phase 1, success metrics, and appendices. Co-Authored-By: Claude Opus 4.8 --- ...2026-06-19-diffprompt-v2-program-design.md | 21 ++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-06-19-diffprompt-v2-program-design.md b/docs/superpowers/specs/2026-06-19-diffprompt-v2-program-design.md index 1ddbbb8..ff57bf6 100644 --- a/docs/superpowers/specs/2026-06-19-diffprompt-v2-program-design.md +++ b/docs/superpowers/specs/2026-06-19-diffprompt-v2-program-design.md @@ -143,15 +143,25 @@ Emerged during product design; the agentic-reliability flagship and the stronges - **Output**: trajectory timeline, root-cause pin, replay-from-step, "add as a regression test". - **Risks**: framework coupling (keep the core trajectory model agnostic, adapters per framework); replay fidelity (deterministic tool stubs for replayed steps). +### 6.5 Red-team / security (new pillar) — "can your prompt be broken?" +Squarely on-niche (LLM security; leverages the user's garak experience). Auto-attacks a system prompt and shows the prompts that succeeded. +- **Attack suite**: a curated, versioned library of probes across families — prompt injection / instruction override, system-prompt extraction, role-play / DAN, encoding & obfuscation (base64 / leetspeak / many-shot), data exfiltration — mapped to **OWASP LLM Top 10** and **garak** probe families. Pluggable so new attacks drop in. +- **Run**: fire each attack against the system prompt via the provider; a specialized security-judge decides success (did it comply / leak / break policy?). Reuses the runner; the ADVERSARIAL generator bucket seeds novel variants. +- **Output (evidence-first)**: resilience score (blocked / total) + by-family breakdown; and the headline — the **verbatim attack inputs that succeeded plus the exact leaked or compliant output**, severity-ranked (critical = prompt extraction / data exfil). +- **Harden**: auto-generate a hardened system prompt (refuse meta-requests about instructions, pin identity, treat retrieved/user text as data) → re-test → projected resilience. Reuses the Fix loop. +- **CI gate**: pin attacks as an adversarial golden set; block merge on any CRITICAL success — a security regression guard. +- **CLI**: `diffprompt redteam [--suite owasp|garak|all]`. +- **Risks**: suite freshness (jailbreaks evolve — version it, allow community probes); security-judge accuracy (false negatives are dangerous — bias toward flagging, require human confirmation on criticals). + ## 7. Product surfaces & UX **Design principle — evidence over abstraction.** Every screen ends in evidence or an action: see the behavior → find the cause → apply a fix → re-verify. Abstract dashboards (radar/bubble charts) are demoted to a secondary "profile" view; the spine is the actual outputs, the actual prompt sentence, the actual trajectory step. (This was an explicit course-correction during design — early chart-only mockups were rejected as "too abstract.") **Entry — two ways in** (`diffprompt` composer, mirrors the CLI): -- **Prompt** — paste v1/v2 (or load from a file / GitHub PR), choose provider · model · judge · test source (auto-generate N *or* upload real `.jsonl`), Run. → feeds Diff / Fix / Drift. +- **Prompt** — paste v1/v2 (or load from a file / GitHub PR), choose provider · model · judge · test source (auto-generate N *or* upload real `.jsonl`), Run. → feeds Diff / Fix / Drift / Red-team. - **Agent** — not pasted; instrument once (`@trace` / OTLP / LangGraph callback) or drop a `trace.json`. → feeds Agent / Provenance / Trace. -**Six surfaces, one engine** (they share the trajectory model, context-provenance, and counterfactual replay): +**Seven surfaces, one engine** (they share the trajectory model, context-provenance, and counterfactual replay): | Surface | Job | Signature element | |---|---|---| @@ -161,6 +171,7 @@ Emerged during product design; the agentic-reliability flagship and the stronges | **Trace** | where did time/money go; what failed quietly | OTel run waterfall + cost/tokens/reliability + Langfuse/Phoenix export | | **Agent** | where did the agent go wrong | trajectory timeline + root-cause step + replay-from-step | | **Provenance** | why did it do that / was it hijacked | per-source influence + trust + verbatim injected text + counterfactual | +| **Red-team** | can your prompt be broken? | resilience score + the verbatim jailbreaks that succeeded + auto-hardened prompt + CI gate | **Design system (RESOLVED — matches the portfolio, warm editorial; not the earlier dark/gold).** Cream paper `#f3ecdf`, rust/terracotta accent `#b5532f`, high-contrast serif display (Fraunces) with rust-italic emphasis, JetBrains Mono for data/code, black pill buttons, label/value cards with thin dividers, a dark mono ticker with `✦` separators, subtle outlined deco circles. Reference mockups (entry + all six surfaces) live in `.superpowers/brainstorm/`. @@ -179,6 +190,7 @@ diffprompt drift check diffprompt watch diffprompt provenance # Idea 3 diffprompt agent # agent debugger (trajectory + root cause + replay) +diffprompt redteam # jailbreak/injection suite + auto-harden diffprompt trace # observability waterfall for a run diffprompt serve # web + API (the composer + six surfaces) diffprompt replay # cache replay @@ -192,7 +204,7 @@ Consistent flags across commands: `--model`, `--judge`, `--provider`, `--local-o ## 11. Phasing (build order; all speced deep, sequenced for delivery) - **Phase 0 — Foundation**: providers, retry, logging, streaming orchestrator, telemetry, caching, test infra + CI gates. *Everything else depends on this.* -- **Phase 1 — Attribution**: pure analysis on the new engine; fastest standalone win after foundation. +- **Phase 1 — Attribution + Red-team**: pure analysis on the new engine, no service infra — the fastest standalone wins after foundation. Red-team is also the strongest near-term niche signal (LLM security). - **Phase 2 — Drift**: store + scheduler + worker pool + alerting + first web views (the observability hero). - **Phase 3 — Provenance + Agent debugger**: source tagging + ablation + trust model + LangGraph adapter; trajectory capture, root-cause detection, replay-from-step. These two share the counterfactual machinery, so they ship together. - **Phase 4 — Web maturation**: the composer entry + unify all six surfaces in the dashboard, Docker self-host, polish. @@ -202,6 +214,7 @@ Consistent flags across commands: `--model`, `--judge`, `--provider`, `--local-o - Reliability: 0 unhandled exceptions across a 100-run live soak; retry recovers ≥ 95% of transient failures. - Accuracy: judge agreement ≥ 0.8 on the eval set; attribution efficiency-axiom error < 5%. - Quality: coverage ≥ 85%; ruff + mypy clean in CI. +- Security: 0 CRITICAL jailbreaks pass the bundled suite against the default hardened template; the red-team CI gate catches a planted regression. - Traction: stars, and at least one external user running drift monitoring. ## 13. Risks (program-level) @@ -215,6 +228,7 @@ Consistent flags across commands: `--model`, `--judge`, `--provider`, `--local-o ## Appendix A — mapping to `future-ideas/IDEAS.md` - Idea 1 → §6.1 Attribution. Idea 2 → §6.2 Drift. Idea 3 → §6.3 Provenance. The repo's own notes (Idea 2 fastest to users, Idea 1 deepest interview story, Idea 3 most unique) are reflected in the phasing rationale. - §6.4 **Agent debugger** is a new pillar added during product design — not in the original IDEAS.md. It composes Provenance + the counterfactual into a trajectory debugger and is the clearest single expression of the agentic-reliability niche. +- §6.5 **Red-team** is likewise a new pillar — auto jailbreak/injection testing + hardening, the clearest expression of the LLM-security niche and a natural fit with the user's garak work. ## Appendix B — validated mockups Interactive mockups built and reviewed via the brainstorming visual companion (warm-editorial), persisted in `.superpowers/brainstorm/`: @@ -224,3 +238,4 @@ Interactive mockups built and reviewed via the brainstorming visual companion (w - `observability-trace.html` — OTel run waterfall + cost/reliability + Langfuse export - `agent-debugger.html` — trajectory timeline + root-cause + replay - `provenance-trace.html` — per-source influence + injected text + counterfactual +- `security-redteam.html` — resilience score + the jailbreaks that succeeded + auto-harden + CI gate