Keep this open on a second screen during Act 4 (labeling). In the annotation
queue, give each example this correctness score, then Add to Reference Dataset.
- 1 = correct, 0 = incorrect
- The UI may reorder examples. Match on the
example_idfield in each example's input JSON — it equals the # column below. - 12 cases · 6 correct / 6 incorrect (balanced, and the gold labels alternate).
- This is the curated demo subset (
DEMO_IDSindata/cases.py); the bank holds alternates if you want to swap a bucket or lengthen the demo.
| # | id | Question (short) | Score | Why (bucket → fixed by) |
|---|---|---|---|---|
| 1 | fact-retention | Free plan deleted-project retention | 0 | fact · says 90 days, truth is 30 → reference |
| 2 | terse-darkmode | enable dark mode | 1 | terse · "Settings > Appearance" is right, just brief → ignore length |
| 3 | complete-delete | permanently delete account | 0 | complete · omits required email confirmation → completeness rule |
| 4 | ok-invite | invite a teammate | 1 | ok · full + correct (control) |
| 5 | extra-reset | reset my password | 0 | extra · invents an SMS code → grounding rule |
| 6 | terse-search | search keyboard shortcut | 1 | terse · "Cmd/Ctrl+K" is right, just brief → ignore length |
| 7 | fact-seats | guests on Team plan | 0 | fact · "unlimited" is wrong (10) → reference |
| 8 | ok-duedate | add a due date | 1 | ok · full + correct (control) |
| 9 | complete-transfer | transfer ownership | 0 | complete · omits required owner approval → completeness rule |
| 10 | ok-mute | mute project notifications | 1 | ok · full + correct (control) |
| 11 | bad-encryption | data encrypted at rest? | 0 | bad · contradicts; says "no" (control) |
| 12 | ok-mfa | turn on two-factor login | 1 | ok · correct, MFA = two-factor (control) |
Tip: the most instructive ones to read aloud are #1 (confident but wrong) and #2 (clearly right, but the biased judge will reject it for being terse). Those two flip together when you add the reference + "facts not style" rule.