engine v0.2: real data-flow graphs + operation-labeled clone detection - #1
Open
M0nkeyFl0wer wants to merge 6 commits into
Open
engine v0.2: real data-flow graphs + operation-labeled clone detection#1M0nkeyFl0wer wants to merge 6 commits into
M0nkeyFl0wer wants to merge 6 commits into
Conversation
The v0.1 data-flow graph lost most edges: call arguments only created edges when they were normalizer temporaries, so y = f(x) produced no x -> y edge, and running the tool on its own analyzer gave 144 nodes / 23 edges / 121 components. The duplication check then used unlabeled isomorphism on those graphs, which flagged unrelated same-shape functions as copy-paste (including this repo's own summary() vs _check_circular_dependencies()). Normalizer: - every Name an operation reads becomes a dependency (args, kwargs, attribute bases, comparison/boolean operands) - control flow contributes statements: if/while tests, for iterables, with contexts, try bodies, aug-assign, comprehensions, unpacking - statements carry category (operation kind) and label (operation identity) plus recorded callee names; locals are scoped per function - inline suppressions: # codetopo: ignore[type] on a def line Analyzer: - structural_duplication now requires category-matched isomorphism AND a consistent bijective renaming between operation labels (type-2 clone); shape-only matches downgrade to advisory structural_similarity - statement_multitask renamed variable_fanout, demoted to advisory, parameters exempt; abstraction_bloat removed (unfalsifiable); isolated_component now per-function; persistent homology opt-in and labeled experimental - thresholds recalibrated for zero errors on micrograd/nanoGPT/minGPT Call graph is built from recorded callees (including inside comprehensions) with ambiguous basenames skipped, instead of string parsing. visit_If/visit_While were genuine clones the new check caught in this repo - now one shared handler. Tests cover the send-criteria guarantees: real edges, control-flow capture, renamed-clone detection, shape-coincidence non-errors, cycle suppression, and a blocking self-check regression test.
nanochat was labeled 'agent-assisted'. Karpathy stated publicly that nanochat is 'basically entirely hand-written (with tab autocomplete)' (x.com/karpathy/status/1977755427569111362), so the hand-vs-agent gradient claim built on that label is withdrawn. The corpus is reframed as what it actually supports: a false-positive-rate measurement on 65 files of known-clean code. Result with the v0.2 engine at default thresholds: 0 errors on micrograd/nanoGPT/minGPT; 3 errors on nanochat, all true positives (three verbatim copy-pasted test methods in tests/test_attention_fallback.py, differing only in tensor sizes). scripts/build_corpus.py regenerates every number, records repo SHAs, and normalizes per file. benchmarks/ adds seeded-defect comparison against pylint (duplicate-code, cyclic-import, too-many-statements) and radon: codetopo uniquely catches same-file exact and renamed clones, function-level call cycles, and straight-line god functions; pylint uniquely catches cross-file exact clones and import cycles. Old v0.1-engine tiers deleted; fix_pass/ab_review dirs kept as historical records and marked as such in the manifest.
README rewritten against docs/send-criteria.md (the reverse-engineered list of what must be true before this is worth sending to a skeptical expert): - 'algebraic topology' headline dropped: the working checks are classical graph theory; persistent homology is opt-in, experimental, and claim-free - 'Goodhart-proof' removed and explicitly disclaimed: determinism buys auditability, not Goodhart-resistance - calibration presented as false-positive rate on hand-written code, with the nanochat correction stated in the open rather than silently fixed; the hand-vs-agent gradient marked as an open question needing a paired corpus - baseline comparison table (generated by benchmarks/) replaces unsupported '10-25x' claims CI self-check is now blocking - a linter whose own CI ignores its output is self-refuting - and the benchmark table is checked for drift. /review skill and viz About text updated to the v0.2 finding taxonomy.
docs/specs/ holds one spec per feature — problem, design rationale, exact interface, failure modes, and a 'done means' checklist — plus a priority-ordered plan. Every spec inherits the send-criteria constraints: no ERROR severity without a measured false-positive rate, every reported number regenerable by a committed script, deterministic checks only. 1. Cross-file structural index & reuse recommender 2. PreToolUse delta gate (block edits that INTRODUCE errors) 3. Hodge architecture metrics (circularity score + implied layering) 4. Persistence fingerprint & drift (stable structural change tracking) 5. Research directions: directed topology for async deadlocks, sheaf cohomology for cross-module consistency — problem statements only, with an explicit promotion rule so the tool never claims them.
…etrics, fingerprints Spec 1 — index.py: per-function labeled subgraphs serialized with cheap invariants for prefiltering. `codetopo index` builds it; `codetopo check --index` adds cross_file_duplication (ERROR — every corpus hit human-verified, see spec) and cross_file_similarity (advisory); `codetopo similar -` reads a draft function on stdin and answers 'this already exists as X' with deterministic, auditable scores (exit 2 on an exact structural duplicate, for hooks). Corpus validation: 0 findings on micrograd/nanoGPT/nanochat; 1 true positive on minGPT (get_config copy-pasted between projects/adder and projects/chargpt). Spec 2 — gate.py + .claude/hooks/codetopo-gate.sh: PreToolUse hook that blocks Write/Edit only when the proposed content would INTRODUCE new error-level findings vs. the file on disk. Pre-existing errors and warnings never block; unparseable/broken/non-Python input fails open. Live in this repo's .claude/settings.json. Spec 3 — hodge.py: combinatorial Hodge decomposition of call-graph net flow. `codetopo arch` reports an inconsistency score in [0,1] (micrograd 0.000, nanoGPT 0.000, minGPT 0.019, nanochat 0.048, this repo 0.089), the implied function layering, and the calls most violating it. Dashboard metric only, never a finding. Spec 4 — fingerprint.py: per-file H0/H1 persistence diagrams + invariants; `codetopo fingerprint`/`codetopo drift` with Hungarian W1 diagram distance. Verified: renaming every identifier gives exactly zero drift; adding functions/cycles moves it. topology/persistence.py shims the PH backend to gudhi (ripser's dependency chain no longer builds: unmaintained hopcroftkarp sdist); ripser still preferred if importable, extra switched to gudhi. Also: isolated_component nodes sorted for deterministic reports; 18 new unit tests (38 total) covering each spec's done-means list.
…d benchmarks Benchmark table: both cross-file rows flip to caught with the index — the renamed cross-file clone is now caught by codetopo alone in the Python ecosystem; module import cycles remain pylint's row and the README says so. Corpus manifest gains cross_file results (the one true-positive minGPT pair) and arch inconsistency reference scores per repo. README documents the proactive loop (reuse recommender, delta gate, enforced budgets), the dashboard layer (arch, fingerprint/drift, with their honesty notes), and relocates 'where the topology actually lives' to the layer where the math is load-bearing.
M0nkeyFl0wer
force-pushed
the
claude/karpathy-assessment-vz70qu
branch
from
July 10, 2026 13:52
04f56f9 to
83b7078
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v0.1 data-flow graph lost most edges: call arguments only created
edges when they were normalizer temporaries, so y = f(x) produced no
x -> y edge, and running the tool on its own analyzer gave 144 nodes /
23 edges / 121 components. The duplication check then used unlabeled
isomorphism on those graphs, which flagged unrelated same-shape
functions as copy-paste (including this repo's own summary() vs
_check_circular_dependencies()).
Normalizer:
attribute bases, comparison/boolean operands)
with contexts, try bodies, aug-assign, comprehensions, unpacking
identity) plus recorded callee names; locals are scoped per function
Analyzer:
a consistent bijective renaming between operation labels (type-2
clone); shape-only matches downgrade to advisory
structural_similarity
parameters exempt; abstraction_bloat removed (unfalsifiable);
isolated_component now per-function; persistent homology opt-in and
labeled experimental
Call graph is built from recorded callees (including inside
comprehensions) with ambiguous basenames skipped, instead of string
parsing. visit_If/visit_While were genuine clones the new check caught
in this repo - now one shared handler.
Tests cover the send-criteria guarantees: real edges, control-flow
capture, renamed-clone detection, shape-coincidence non-errors, cycle
suppression, and a blocking self-check regression test.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01BHa7G47j5nXSzs2ihs4Kkj