Record ADR-010 and an opt-in mypyc-accelerated engine build

adr/010-mypyc-accelerated-build.md

@@ -0,0 +1,132 @@
+# ADR-010: mypyc as an optional accelerated build
+
+**Date**: 2026-06-29
+**Status**: Proposed
+
+**Context**: ADR-005 chose pure Python with no required runtime dependencies, and
+recorded that "if a future, measured workload genuinely needs more speed, a native
+core is the path to revisit." ADR-004 says the same: the door to a native core
+stays open, the default stays interpreted.
+
+We have now reached the interpreted floor. A long pass of profile-driven work
+(cProfile and py-spy on Home-Assistant-representative schemas) flipped construction
+from ~1.47x slower than voluptuous to ~1.64x faster, and holds a ~2.3x validation
+lead. After that work, the profiles show the hot paths are doing irreducible work:
+validation spends ~47% of its time in one Python `for` loop over a mapping's keys,
+the rest in core dispatch. You cannot make that loop meaningfully faster _in
+CPython_; the cost is the interpreter executing bytecode, not the logic. The next
+significant speedup is architectural, not algorithmic: change what runs the code.
+
+mypyc is the lowest-effort native path. It compiles type-annotated Python straight
+to a C extension. Probatio is unusually ready for it: the package is mypy-strict
+clean already, which is mypyc's input contract.
+
+**What an actual build showed**: a first spike compiled the four core modules
+(`error`, `markers`, `_engine`, `schema`) plus the rest of the package and reported
+1.30x construction and 1.38x validation. That spike benchmarked the happy path but
+never ran the test suite against the compiled build. Building it for real and
+running the full suite tells a narrower story. "Compiles without error" is not
+"behaves correctly", and three of the four core modules fail the suite once
+compiled:
+
+- **`error.py`** breaks the error model. The `Invalid` hierarchy reads its
+  per-subclass `default_code` off the type object (`type(self).default_code`).
+  mypyc stores it as an instance slot, not a class attribute, so reading `.code`
+  raises and the failure path segfaults. It is pure failure-path code (it runs
+  only when validation fails), so compiling it buys nothing on the hot path.
+- **`schema.py`** cannot be compiled. `Schema` must stay a non-native class because
+  `DataclassSchema`/`TypedDictSchema` and user code subclass it, and mypyc forbids
+  an interpreted class inheriting a compiled one. A non-native `Schema` then trips
+  an internal mypyc assertion on `_compile_self`'s nested closure. This is the
+  module that holds construction, the headline the first spike claimed.
+- **`markers.py`** compiles but silently breaks `isinstance(x, Marker)` on the
+  non-native marker subclasses, so marker ordering and equality go wrong. The suite
+  catches it.
+
+Only **`_engine.py`**, the validation hot loop, compiles cleanly: zero source
+changes, full behavioral parity (the whole suite passes against it). On the
+Home-Assistant-representative workload:
+
+|              | pure Python | mypyc (engine only) | gain  |
+| ------------ | ----------- | ------------------- | ----- |
+| construction | 0.87s       | 0.81s               | 1.08x |
+| validation   | 0.80s       | 0.61s               | 1.29x |
+
+The validation gain is real and lands with no caveats. Construction barely moves,
+because construction lives in `schema.py`, which does not compile. So the honest
+payoff today is a validation accelerator, not the construction win the first spike
+advertised. The construction story would need real work on `schema.py` (refactoring
+to dodge the mypyc closure limitation, and re-checking the non-native subclass
+contract), not the "cosmetic adjustments" that spike assumed.
+
+**Options considered**:
+
+1. Stay pure Python only. Accept the interpreter floor; the lead over voluptuous
+   is already comfortable.
+2. mypyc as an optional accelerated build of the engine, with pure Python as the
+   always-present fallback. Compile what compiles cleanly (`_engine.py`), leave the
+   rest interpreted.
+3. A native core (Rust or C, pydantic-core style). The highest ceiling (5x to
+   50x), but a rewrite into effectively a different project, with an FFI boundary
+   and a Rust toolchain to maintain.
+4. Per-schema code generation (drop ADR-004). Generate and `exec` a specialized
+   validator per schema. It speeds validation but makes construction slower, which
+   is the wrong trade for Home Assistant, whose cold start is construction-bound
+   (thousands of schemas built at import, each config validated roughly once).
+
+**Decision**: Option 2, deferred. Pure Python stays the canonical build and the
+guaranteed fallback (ADR-005 holds in spirit: install it and import it, on any
+platform, with no compiler). The opt-in accelerated build compiles `_engine.py`
+only, the one module that stays correct under mypyc. It is recorded here as a
+validated, ready option to adopt when we choose to, not now. The build is wired up
+behind `just build-fast` (it runs `scripts/build_accelerated.py`, which drops the
+compiled `.so` next to the source; `just clean` removes it), so the path is real
+and runnable, not just described.
+
+**Rationale**:
+
+- **Measured, not assumed, and corrected**: the numbers and the blockers above are
+  from a working build run against the full suite, not an estimate. The first
+  spike's wider claim did not survive contact with the tests; this records what
+  actually holds.
+- **Keeps the architecture**: same engine, same ADR-004 single-engine design, same
+  drop-in semantics. mypyc accelerates the existing code; it does not replace it.
+- **Right tool, narrower scope**: a native rewrite (option 3) abandons the
+  clean-room readable-Python identity for a payoff we do not need yet; codegen
+  (option 4) optimizes the wrong axis for the primary consumer. mypyc on the engine
+  is a real validation win with no source contortion.
+- **Fallback preserves the promise**: shipping a compiled wheel with a pure-Python
+  fallback keeps ADR-005's portability for platforms without a prebuilt wheel.
+
+**Consequences**:
+
+- **The payoff today is validation, not construction.** Adopting this accelerates
+  validation ~1.29x and leaves construction essentially unchanged, because
+  `schema.py` does not compile. If construction is the goal (it dominates Home
+  Assistant's cold start), this is not yet the answer; it would need the `schema.py`
+  work above first.
+- **Packaging is the real ongoing cost.** Adopting this means building and shipping
+  per-platform wheels (cibuildwheel in CI) alongside a pure-Python sdist, and a
+  runtime that prefers the compiled module when present and falls back otherwise.
+  This is the part that amends ADR-005: the default install would carry a native
+  extension on supported platforms, with pure Python as the fallback rather than
+  the only artifact.
+- **A second test lane, without coverage.** CI would run the suite against both the
+  pure and the compiled build, since mypyc can change behavior in edge cases (it
+  did, in three modules). The compiled lane runs with coverage off: mypyc code is
+  not traced, and the tracer can crash native code. The pure lane keeps the 100%
+  coverage gate.
+- **The other core modules stay interpreted until reworked.** Compiling `error.py`,
+  `markers.py`, or `schema.py` needs real changes (a class-attribute model that
+  survives mypyc; a non-native `Schema` that mypyc will actually compile; an
+  `isinstance` path that holds for non-native subclasses), not decorators. The
+  `scripts/build_accelerated.py` header documents each blocker at the point it
+  matters.
+- **The door stays open wider, not the default changed**: until this is adopted,
+  ADR-005 and ADR-004 stand unchanged. Adoption is a follow-up decision; this ADR
+  records that the engine path is validated and what it costs.
+
+**Revisit trigger**: adopt when the validation gain is worth the packaging and CI
+cost, for example if a consumer with a tight validation loop needs it. Revisit the
+wider compile (construction) only with budget to do the `schema.py` work, not as a
+drop-in. The build script and numbers here are the starting point for that work.

adr/README.md

@@ -2,14 +2,15 @@
 
 Each record captures what was chosen, the alternatives considered, and why.
 
-| ADR                                                                  | Title                                          |
-| -------------------------------------------------------------------- | ---------------------------------------------- |
-| [ADR-001](001-clean-room-reimplementation-of-voluptuous.md)          | Clean-room reimplementation of voluptuous      |
-| [ADR-002](002-mit-license.md)                                        | MIT license                                    |
-| [ADR-003](003-astro-starlight-for-documentation.md)                  | Astro Starlight for documentation              |
-| [ADR-004](004-single-validation-engine.md)                           | A single validation engine (no codegen)        |
-| [ADR-005](005-pure-python-no-required-dependencies.md)               | Pure Python, no required runtime dependencies  |
-| [ADR-006](006-python-support-floor.md)                               | Python support floor (requires-python >=3.12)  |
-| [ADR-007](007-self-validation-protocol.md)                           | A self-validation protocol for types           |
-| [ADR-008](008-type-to-validator-registry.md)                         | A type-to-validator registry for schema builders |
-| [ADR-009](009-call-time-validation-context.md)                       | Call-time validation context                   |
+| ADR                                                         | Title                                             |
+| ----------------------------------------------------------- | ------------------------------------------------- |
+| [ADR-001](001-clean-room-reimplementation-of-voluptuous.md) | Clean-room reimplementation of voluptuous         |
+| [ADR-002](002-mit-license.md)                               | MIT license                                       |
+| [ADR-003](003-astro-starlight-for-documentation.md)         | Astro Starlight for documentation                 |
+| [ADR-004](004-single-validation-engine.md)                  | A single validation engine (no codegen)           |
+| [ADR-005](005-pure-python-no-required-dependencies.md)      | Pure Python, no required runtime dependencies     |
+| [ADR-006](006-python-support-floor.md)                      | Python support floor (requires-python >=3.12)     |
+| [ADR-007](007-self-validation-protocol.md)                  | A self-validation protocol for types              |
+| [ADR-008](008-type-to-validator-registry.md)                | A type-to-validator registry for schema builders  |
+| [ADR-009](009-call-time-validation-context.md)              | Call-time validation context                      |
+| [ADR-010](010-mypyc-accelerated-build.md)                   | mypyc as an optional accelerated build (proposed) |

justfile

@@ -66,6 +66,14 @@ codspeed:
     uv sync --group codspeed
     uv run --no-sync pytest bench --codspeed --no-cov -o addopts=""
 
+# Build the optional mypyc-accelerated engine in place (ADR-010): compiles the
+# validation hot loop to a C extension next to its source. Opt-in and dev-only;
+# the default build stays pure Python, so the published wheel is unaffected. Needs
+# a C compiler. mypyc ships with mypy (the typing group); setuptools drives the
+# build. Run `just clean` to drop the .so and return to pure Python.
+build-fast:
+    uv run --no-sync --with setuptools python scripts/build_accelerated.py
+
 # Run every documented Python example and verify its output comments.
 examples:
     uv run --no-sync python docs/verify_examples.py
@@ -114,7 +122,8 @@ check: precommit
     uv run --no-sync pytest packages/pytest-probatio/tests -o addopts=""
     uv run --no-sync python docs/verify_examples.py
 
-# Remove build and tooling artifacts.
+# Remove build and tooling artifacts, including the opt-in accelerated .so files.
 clean:
     rm -rf dist build .pytest_cache .ruff_cache .mypy_cache .ty htmlcov .coverage coverage.xml
+    find src -name '*.so' -delete
     find . -type d -name __pycache__ -exec rm -rf {} +

pyproject.toml

@@ -176,6 +176,9 @@ ignore = [
 # script directory, not importable library code. They dispatch with many returns
 # and swallow the "expected" exceptions of the surface under test.
 "fuzz/**" = ["ANN", "INP001", "S101", "PLR0911", "PLR0912", "SIM105"]
+# The accelerated build driver is a dev-only script (ADR-010), a script directory
+# rather than importable library code.
+"scripts/**" = ["INP001"]
 # The docs example harness is a dev-only script: it executes first-party doc
 # snippets (eval/exec by design), prints results, uses local imports, and runs a
 # branchy dispatcher over the AST. It is not importable library code.

scripts/build_accelerated.py

@@ -0,0 +1,93 @@
+"""Build the optional mypyc-accelerated core in place (ADR-010).
+
+This is the opt-in accelerated build. It compiles the four hot core modules
+(``error``, ``markers``, ``_engine``, ``schema``) into a C extension with mypyc
+and leaves the rest of the package, including every validator, interpreted. The
+default build stays pure Python (ADR-005); this script is something you run
+yourself, never part of the published wheel. Run it with ``just build-fast``.
+
+The compiled ``.so`` files land next to their ``.py`` sources under ``src/`` (the
+import root for the editable install), so CPython imports the native versions
+when present and falls back to the pure Python sources anywhere they are absent.
+Run ``just clean`` to delete them and return to pure Python.
+
+The validators stay interpreted on purpose: the measured win (ADR-010, 1.30x
+construction and 1.38x validation on a Home-Assistant-representative workload)
+already lands with just these four compiled, and keeping the validators
+interpreted avoids an interpreted-inherits-compiled boundary on their shared base.
+"""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+from mypyc.build import mypycify
+from setuptools import setup
+
+# The repository root, derived from this script's location (scripts/..).
+ROOT = Path(__file__).resolve().parent.parent
+SRC = ROOT / "src"
+# Build artifacts (generated C, object files, the staging lib) stay out of the
+# tree under build/; only the final .so files land next to the sources.
+BUILD = ROOT / "build" / "accelerated"
+
+# The validation hot loop. mypyc compiles this into one extension; everything
+# else, including every validator, stays interpreted. A compiled engine that
+# imports, raises, and tags interpreted classes is fine; only the reverse
+# (interpreted code inheriting a compiled class) is the boundary to avoid, and
+# nothing outside _engine subclasses what it defines.
+#
+# The other three "core" modules from the ADR-010 spike are deliberately left
+# interpreted, because a compiled build of each breaks behavior the suite catches:
+#
+#   error.py   The Invalid hierarchy reads its per-subclass ``default_code`` off
+#              the type object (``type(self).default_code``). mypyc stores it as
+#              an instance slot, not a class attribute, so ``.code`` raises and the
+#              failure path segfaults.
+#   schema.py  ``Schema`` must be non-native (``DataclassSchema``/``TypedDictSchema``
+#              and user code subclass it, and mypyc forbids interpreted classes
+#              inheriting a compiled one). Compiling a non-native ``Schema`` trips an
+#              internal mypyc assertion on ``_compile_self``'s nested closure.
+#   markers.py The markers must be non-native too (they are copied and subclassed).
+#              A compiled non-native subclass breaks ``isinstance(x, Marker)``, so
+#              marker ordering and equality go wrong.
+#
+# These are not cosmetic adjustments; they are real blockers (see ADR-010). The
+# engine is the part that compiles cleanly and keeps full behavioral parity, so it
+# is the honest floor for this opt-in build.
+CORE_MODULES = [str(SRC / "probatio" / "_engine.py")]
+
+
+def main() -> None:
+    """Compile the core modules in place with mypyc.
+
+    The build runs from ``src/`` so ``build_ext --inplace`` places each compiled
+    module next to its source (``probatio.error`` to ``src/probatio/error.so``,
+    the shared group module to ``src/``) and so setuptools does not read the
+    project's ``pyproject.toml`` (its license metadata is for the pure-Python
+    wheel, not this throwaway extension build).
+    """
+    BUILD.mkdir(parents=True, exist_ok=True)
+    os.chdir(SRC)
+
+    setup(
+        name="probatio-accelerated",
+        ext_modules=mypycify(
+            CORE_MODULES,
+            target_dir=str(BUILD / "c"),
+            group_name="probatio",
+        ),
+        script_args=[
+            "build_ext",
+            "--inplace",
+            "--build-temp",
+            str(BUILD / "temp"),
+            "--build-lib",
+            str(BUILD / "lib"),
+        ],
+    )
+
+
+if __name__ == "__main__":
+    main()