About two-thirds of the instruction files developers write for AI coding agents contain nothing a record of the agent's actions could check. Across all files, under 1 in 10 individual rules is verifiable.
Measured on 4,031 public CLAUDE.md files, weighted to GitHub's real file-size distribution, with
the central judgement made by six independent raters working across two separately-worded rubrics
(Fleiss κ = 0.752).
There is an active and unresolved argument about whether agent instruction files do anything. "AGENTS.md outperforms skills in our agent evals", "Evaluating AGENTS.md: are they helpful for coding agents?", "A good AGENTS.md is a model upgrade. A bad one is worse than no docs at all" — each drew hundreds of comments, and they do not agree with one another.
Meanwhile developers keep reporting that the agent ignores the file. CLAUDE.md drew over 1,200
Hacker News comments in 2026 with no decline through the year, and an Anthropic issue titled
"[BUG] Claude Code continually ignores CLAUDE.MD file" was closed as not planned, labelled
area:model.
The argument is mostly conducted without data, and it skips a prior question. Before asking does the agent follow your rules, it is worth asking which of your rules could anyone ever check?
That question has a precise form. A Claude Code session leaves a transcript recording every shell command the agent ran and every file it read, created, or modified. It does not record what the code said, what an edit contained, or whether a command succeeded. So:
Given a log of commands run and paths touched, could you decide whether this rule was followed?
We built a tool to do exactly that — and then measured how much of a real CLAUDE.md it could possibly apply to.
| Bound | Share of rule-like lines |
|---|---|
| Unanimous across all raters | 3.3% |
| Majority | 8.3% |
| Best estimate | ~6–9% |
| Most generous defensible reading | 11.7% |
Translated into whole files, weighted by GitHub's real size distribution:
| Share of all CLAUDE.md files | |
|---|---|
| Contain ≥1 checkable rule (the ceiling for any tool) | 30–35% |
| What our extractor finds today | 7.0% |
Roughly two-thirds of CLAUDE.md files contain nothing a transcript could ever verify.
The recurring pattern is rules that look command-shaped without being about commands:
- "Never use
shell=Truewith subprocess" — about code being written, not a command run. - "Never embed CSS strings in TypeScript files" — the log shows a
.tsfile was edited, never what the edit contained. - "All endpoints return
ResponseDTO<T>— never raw DTOs" — pure code content. - "Use descriptive variable names" — no observable event at all.
A transcript records which commands ran and which files changed. Most CLAUDE.md rules are about what the code says. That is the whole finding, and no amount of engineering changes it.
By subject, the rules people actually write:
| Subject | Share of rules |
|---|---|
| file path / glob | 17% |
| tests | 10% |
| command / CLI | 10% |
| types / typing | 7% |
| commit / PR / git | 6% |
| code style / naming | 5% |
Even the top categories are mostly unreachable: a rule about a path is checkable, but a rule about what belongs in the file at that path is not.
Real examples all six raters agreed on:
- "Always run
npm run lint && npm run typecheck && npm testbefore considering a change complete." - "Test file must be in the same directory as the source file."
- "Never edit
vendor/lq-ai."
They share a shape: a named command or a named path, with a trigger that is itself visible in the log.
For a meaningful share of even the checkable rules, what gets verified is a proxy. This surfaced when two raters using a differently-worded rubric refused rules the first panel had accepted unanimously:
| Rule | Why the stricter reading refused it |
|---|---|
| "Only commit when tests are green" | The log shows the test ran, never its exit code |
"If you change the API, update BUILD.bazel" |
You see edits, not whether the API changed |
"ALL CSS must be in .scss under src/styles/" |
A misplaced file is visible; CSS inlined elsewhere is not |
A tool can honestly report "a test command ran before the commit." It cannot report "the tests passed." Anything claiming the latter from a transcript is overclaiming.
Corpus. 4,031 public CLAUDE.md files via GitHub code search, faceted across twelve size buckets
from <400 B to >25 kB, one file per repository, deduplicated by content hash. A separate corpus
of 400 AGENTS.md files behaves identically (1.9% extractable vs 1.2%), so this is not a
Claude-specific artifact.
Weighting. A single search query returns at most 1,000 results, so coverage requires faceting — which makes the raw sample's size mix an artifact of collection. This matters more than anything else here, because rule density scales ~100× with file size:
| File size | Rule-like lines per file |
|---|---|
| <400 B | 0.44 |
| 1.8–2.6 kB | 3.44 |
| 5–7 kB | 7.87 |
| >25 kB | 43.93 |
Every prevalence figure is therefore weighted by GitHub's reported population count per bucket, with the sample covering 100% of the population by size.
Ceiling. 120 rule-like lines drawn reproducibly (sorted by content hash) from both corpora, labelled independently by six raters — four on one rubric, two on a rubric rewritten from scratch as a robustness check. No rater could see another's answers.
| Panel | Raters | Mean checkable |
|---|---|---|
| Original rubric | 4 | 8.75% |
| Variant rubric | 2 | 6.67% |
| All six | 6 | 8.06% |
Pairwise Cohen's κ 0.78–0.95 within the first panel; Fleiss κ = 0.861 within it, and 0.752 across both rubrics. Rewording moved the estimate about 2 points, which is why the headline is a range.
The headline number moved four times during this work, always downward, and every move came from questioning a measurement rather than gathering more data. The corrections are the most useful part of the method:
55.7% → wrong. A regex heuristic asked "does the rule name a command or a path?" It fired on any prose containing a backtick. It also produced a ranked "spec" for which matchers to build next — a spec that would have had us build matchers for unverifiable prose.
18% → wrong. Hand-labelling, but 60 lines by a single labeller — the same person who built the tool the number justified — using a CHECKABLE/CHECKABLE-WEAK split. The weak tier evaporated under independent labelling. Tellingly, the old strong-only tier was 5.0%, close to the eventual unanimous 6.7%.
8.75% → too precise. A single rubric, whose worked examples were doing part of the work. Removing them moved six raters to 6.67%.
Prevalence read 11.0%, then 4.6%, then 7.0%. The first two were artifacts of which size buckets happened to be sampled, before population weighting.
Two lessons generalise. Never let a tool's author be the only rater of the tool's own justification. And the sampling frame quietly encodes an answer — check the frame before trusting the number.
An appealing idea fell out of this: detect CLAUDE.md rules that contradict each other — "Use
pnpm, never npm" beside "Always run npm test before committing". It needs no transcript, so
it escapes the ceiling entirely.
Implemented and run across all 4,031 files: zero found. A contradiction needs two rules, and only 50 files in 4,031 yield two extractable rules. Pairwise reasoning inherits the same extraction bottleneck. Recorded here because the idea is attractive enough that someone will have it again.
Worth noting that the problem is real even though we could not detect it. Anthropic's documentation warns about it explicitly — "if two rules contradict each other, Claude may pick one arbitrarily" — and advises reviewing your files periodically to remove conflicting instructions. Our zero is a statement about detection at achievable extraction density, not evidence that contradictions don't occur.
All six raters are LLM instances. They may share a systematic blind spot a human panel would not, and their independence is procedural, not absolute. Varying the rubric is a partial mitigation, not a substitute. This is the most significant limitation and the one most likely to change the number.
The sample is relevance-ranked within each size bucket. Weighting corrects the size mix, not GitHub's ranking bias. Population counts are search estimates and drift.
n = 120 for the ceiling. The 95% interval is roughly ±5 points, and the rubric itself moves the estimate ~2 points.
"Checkable" is defined against one specific evidence model — commands run and paths touched. A richer log (exit codes, diffs) would move the ceiling up. That is a real design question, not a fixed fact about the universe.
For anyone writing a CLAUDE.md. The rules with any hope of being enforced are the ones naming a command or a path. Everything else is a request — and Anthropic's own documentation says so plainly:
"Both are loaded at the start of every conversation. Claude treats them as context, not enforced configuration. To block an action regardless of what Claude decides, use a PreToolUse hook instead."
"CLAUDE.md content is delivered as a user message after the system prompt, not as part of the system prompt itself. Claude reads it and tries to follow it, but there's no guarantee of strict compliance, especially for vague or conflicting instructions."
"Settings rules are enforced by the client regardless of what Claude decides to do. CLAUDE.md instructions shape Claude's behavior but are not a hard enforcement layer."
The gap this study measures is therefore not a surprise to the vendor. What is new is the size of it: not merely that instructions are unenforced, but that most of them could not be checked even after the fact, by anyone.
For anyone building a tool. The checkable subset is also precisely the subset a PreToolUse hook
can block before it happens. That is the hard question for any after-the-fact compliance tool,
and we do not have a satisfying answer to it.
For the "do these files work?" argument. Both sides are partly measuring something unmeasurable. When ~90% of a file's rules cannot be verified even in principle, "did the agent follow it?" is not one question, and evaluations that treat it as one will keep disagreeing.
Everything behind every number is here. The code is measurement apparatus — how the 7.0% figure was produced, not a product being offered.
npm install
npm test # 347 tests
npm run typecheck| Path | What it is |
|---|---|
scripts/corpus/collect.mjs |
Collects a CLAUDE.md corpus from GitHub code search |
scripts/corpus/analyze.ts |
Prevalence, cue and subject frequencies |
scripts/corpus/weighted.ts |
Population-weighted prevalence, using GitHub's per-bucket counts as weights |
scripts/corpus/gate-ceiling.ts |
Share of files that could ever yield a finding |
scripts/corpus/conflicts.ts |
Contradictory-rule detection |
scripts/corpus/labels/ |
The exact 120-line sample and all six independent label sets |
tool/ |
The extractor and checkers that produced the "found today" numbers |
Point the analysis scripts at a collected corpus with CORPUS_DIR. The corpus itself is not
committed — those are third-party files — and GitHub's ranking is not stable, so a fresh collection
will not reproduce this sample exactly.
MIT