🔄 promptdrift

Detect and visualize drift between versions of LLM prompts.

Prompts are untyped, high-entropy configuration. When they silently drift between versions — a variable renamed, a constraint dropped — downstream code breaks and nobody notices until production. promptdrift gives you visibility: line-level diffs, template-variable tracking, and a severity verdict you can gate CI on.

Features

• 🔍 Line + variable diff — see exactly what changed between two prompt versions
• 🧩 Variable drift detection — added/removed template variables are flagged as breaking; renames are not
• 🚦 Severity scoring — breaking / major / minor / trivial / none, with --fail-on-breaking for CI
• 🗃️ Snapshot store — versioned, hash-keyed snapshots in .promptdrift/; watch compares the working file vs the last baseline
• 🎨 Readable output — color-coded unified diff, plus --json for automation
• 🚫 Zero dependencies — pure Python stdlib

Supported variable syntaxes

Syntax	Example	Tag
Jinja2 / Mustache	{{ name }}	jinja
Python str.format	{name}	format
Dollar-brace	${name}	dollar-brace
Bare dollar	$name	dollar
Angle brackets	<<name>>	angle
Double brackets	[[name]]	bracket

Quick Start

pip install promptdrift

Usage

Diff two prompt versions

promptdrift diff prompts/v1.txt prompts/v2.txt

BREAKING  Breaking drift: -variables ['target_language']
Variable drift:
  - target_language  (removed — breaking)

Diff:
--- prompts/v1.txt
+++ prompts/v2.txt
@@ -1,1 +1,1 @@
-You are a translator. Translate {text} into {target_language}.
+You are a translator. Translate {text}.

Gate CI on breaking drift

promptdrift diff old.txt new.txt --fail-on-breaking --json
# exits 1 if a variable was added/removed

List template variables in a prompt

promptdrift variables prompts/sys.txt

Variables in prompts/sys.txt (3):
  name       [jinja]
  question   [format]
  context    [dollar-brace]

Snapshot-based workflow (for watch)

promptdrift snapshot prompts/sys.txt --label "v1 production"
# ...edit the prompt...
promptdrift watch prompts/sys.txt            # compare vs last snapshot
promptdrift history prompts/sys.txt          # show all snapshots

Severity model

Severity	Trigger	CI gate
breaking	a variable was added or removed (not a 1:1 rename)	fails
major	≥40% of lines changed (large rewrite)	passes
minor	10–40% of lines changed	passes
trivial	<10% of lines changed	passes
none	identical	passes

A rename (variable set size preserved, names swapped with token overlap) is not breaking — it's reported but allowed, since the caller can usually adapt the fill key.

Development

git clone https://github.com/alvabillwu/promptdrift.git
cd promptdrift
pip install -e ".[dev]"
pytest -v
python examples/demo.py

License

MIT © alvabillwu