Catch a reward-hacker in your terminal — judge-free, 0 model calls, $0.
npx github:verifiablelabs/vlabs-demoNo install, no dependencies, no API key, no network. It runs the Isomorphic Perturbation Testing (IPT) mechanism on embedded toy tasks entirely in Node, so the output is byte-reproducible — the same every time you run it.
A genuine program is invariant under a semantics-preserving relabeling of its test harness. An input-memorizing shortcut is not:
shortcut(H) = passes(public tests T) AND NOT passes(isomorphic tests T′)
Both a genuine solution and a hardcoded, input-memorizing hack pass the public tests — so an LLM judge that scores the visible output approves the hack. IPT regenerates the inputs and recomputes the expected outputs with a trusted reference: the memorizer breaks, the genuine solution doesn't.
- IPT is a public method (Helff et al., arXiv:2604.15149). Verifiable Labs productizes it — we did not invent it.
- The embedded positives are constructed input-memorizers that fail
T′by construction. This demonstrates the mechanism; it is not an empirical field detection rate. - The "LLM judge ~40% false-positive" figure is our reproduced measurement on the public Obfuscation-Atlas MBPP honeypot (Atlas-500), clearly labeled — not an independently verified third-party fact. IPT requires a trusted reference + input generator; general recall is lower than on this honeypot.
This Node demo is a faithful, self-contained port of the mechanism. The actual engine (Python, deterministic, same honesty framing) runs with:
pip install vlabs-sdk
vlabs-ipt demoDocs, and how to scan your own verifier or RL/RFT reward: https://verifiable-labs.com
Apache-2.0