From 0ccaa184d74b2c96faf813bdf7e11c3cf12d4c23 Mon Sep 17 00:00:00 2001 From: Franck Nijhof Date: Mon, 29 Jun 2026 07:27:01 +0000 Subject: [PATCH] Record ADR-010 and an opt-in mypyc-accelerated engine build Captures the mypyc exploration as a proposed (not adopted) decision, with the actual build code behind it. An earlier spike compiled the four core modules and reported ~1.30x construction and ~1.38x validation, but it never ran the suite against the compiled build. Building it for real and running the full suite tells a narrower story: only the validation engine compiles cleanly. error.py breaks the error model. It reads default_code off the type object, which mypyc stores as an instance slot, so .code raises and the failure path segfaults. schema.py will not compile a non-native Schema, which it must be, since DataclassSchema/TypedDictSchema and user code subclass it. mypyc trips an internal assertion on _compile_self's closure. markers.py compiles but breaks isinstance on the non-native marker subclasses, so marker ordering and equality go wrong. _engine.py, the validation hot loop, compiles with zero source changes and full behavioral parity (the whole suite passes against it): construction 0.87s to 0.81s 1.08x validation 0.80s to 0.61s 1.29x The payoff is validation, not the construction win the spike advertised, because construction lives in schema.py. The opt-in build is wired behind `just build-fast` (scripts/build_accelerated.py drops the compiled .so next to the source; `just clean` removes it). Pure Python stays the default and guaranteed fallback; adoption is deferred. --- adr/010-mypyc-accelerated-build.md | 132 +++++++++++++++++++++++++++++ adr/README.md | 23 ++--- justfile | 11 ++- pyproject.toml | 3 + scripts/build_accelerated.py | 93 ++++++++++++++++++++ 5 files changed, 250 insertions(+), 12 deletions(-) create mode 100644 adr/010-mypyc-accelerated-build.md create mode 100644 scripts/build_accelerated.py diff --git a/adr/010-mypyc-accelerated-build.md b/adr/010-mypyc-accelerated-build.md new file mode 100644 index 0000000..45a9b16 --- /dev/null +++ b/adr/010-mypyc-accelerated-build.md @@ -0,0 +1,132 @@ +# ADR-010: mypyc as an optional accelerated build + +**Date**: 2026-06-29 +**Status**: Proposed + +**Context**: ADR-005 chose pure Python with no required runtime dependencies, and +recorded that "if a future, measured workload genuinely needs more speed, a native +core is the path to revisit." ADR-004 says the same: the door to a native core +stays open, the default stays interpreted. + +We have now reached the interpreted floor. A long pass of profile-driven work +(cProfile and py-spy on Home-Assistant-representative schemas) flipped construction +from ~1.47x slower than voluptuous to ~1.64x faster, and holds a ~2.3x validation +lead. After that work, the profiles show the hot paths are doing irreducible work: +validation spends ~47% of its time in one Python `for` loop over a mapping's keys, +the rest in core dispatch. You cannot make that loop meaningfully faster _in +CPython_; the cost is the interpreter executing bytecode, not the logic. The next +significant speedup is architectural, not algorithmic: change what runs the code. + +mypyc is the lowest-effort native path. It compiles type-annotated Python straight +to a C extension. Probatio is unusually ready for it: the package is mypy-strict +clean already, which is mypyc's input contract. + +**What an actual build showed**: a first spike compiled the four core modules +(`error`, `markers`, `_engine`, `schema`) plus the rest of the package and reported +1.30x construction and 1.38x validation. That spike benchmarked the happy path but +never ran the test suite against the compiled build. Building it for real and +running the full suite tells a narrower story. "Compiles without error" is not +"behaves correctly", and three of the four core modules fail the suite once +compiled: + +- **`error.py`** breaks the error model. The `Invalid` hierarchy reads its + per-subclass `default_code` off the type object (`type(self).default_code`). + mypyc stores it as an instance slot, not a class attribute, so reading `.code` + raises and the failure path segfaults. It is pure failure-path code (it runs + only when validation fails), so compiling it buys nothing on the hot path. +- **`schema.py`** cannot be compiled. `Schema` must stay a non-native class because + `DataclassSchema`/`TypedDictSchema` and user code subclass it, and mypyc forbids + an interpreted class inheriting a compiled one. A non-native `Schema` then trips + an internal mypyc assertion on `_compile_self`'s nested closure. This is the + module that holds construction, the headline the first spike claimed. +- **`markers.py`** compiles but silently breaks `isinstance(x, Marker)` on the + non-native marker subclasses, so marker ordering and equality go wrong. The suite + catches it. + +Only **`_engine.py`**, the validation hot loop, compiles cleanly: zero source +changes, full behavioral parity (the whole suite passes against it). On the +Home-Assistant-representative workload: + +| | pure Python | mypyc (engine only) | gain | +| ------------ | ----------- | ------------------- | ----- | +| construction | 0.87s | 0.81s | 1.08x | +| validation | 0.80s | 0.61s | 1.29x | + +The validation gain is real and lands with no caveats. Construction barely moves, +because construction lives in `schema.py`, which does not compile. So the honest +payoff today is a validation accelerator, not the construction win the first spike +advertised. The construction story would need real work on `schema.py` (refactoring +to dodge the mypyc closure limitation, and re-checking the non-native subclass +contract), not the "cosmetic adjustments" that spike assumed. + +**Options considered**: + +1. Stay pure Python only. Accept the interpreter floor; the lead over voluptuous + is already comfortable. +2. mypyc as an optional accelerated build of the engine, with pure Python as the + always-present fallback. Compile what compiles cleanly (`_engine.py`), leave the + rest interpreted. +3. A native core (Rust or C, pydantic-core style). The highest ceiling (5x to + 50x), but a rewrite into effectively a different project, with an FFI boundary + and a Rust toolchain to maintain. +4. Per-schema code generation (drop ADR-004). Generate and `exec` a specialized + validator per schema. It speeds validation but makes construction slower, which + is the wrong trade for Home Assistant, whose cold start is construction-bound + (thousands of schemas built at import, each config validated roughly once). + +**Decision**: Option 2, deferred. Pure Python stays the canonical build and the +guaranteed fallback (ADR-005 holds in spirit: install it and import it, on any +platform, with no compiler). The opt-in accelerated build compiles `_engine.py` +only, the one module that stays correct under mypyc. It is recorded here as a +validated, ready option to adopt when we choose to, not now. The build is wired up +behind `just build-fast` (it runs `scripts/build_accelerated.py`, which drops the +compiled `.so` next to the source; `just clean` removes it), so the path is real +and runnable, not just described. + +**Rationale**: + +- **Measured, not assumed, and corrected**: the numbers and the blockers above are + from a working build run against the full suite, not an estimate. The first + spike's wider claim did not survive contact with the tests; this records what + actually holds. +- **Keeps the architecture**: same engine, same ADR-004 single-engine design, same + drop-in semantics. mypyc accelerates the existing code; it does not replace it. +- **Right tool, narrower scope**: a native rewrite (option 3) abandons the + clean-room readable-Python identity for a payoff we do not need yet; codegen + (option 4) optimizes the wrong axis for the primary consumer. mypyc on the engine + is a real validation win with no source contortion. +- **Fallback preserves the promise**: shipping a compiled wheel with a pure-Python + fallback keeps ADR-005's portability for platforms without a prebuilt wheel. + +**Consequences**: + +- **The payoff today is validation, not construction.** Adopting this accelerates + validation ~1.29x and leaves construction essentially unchanged, because + `schema.py` does not compile. If construction is the goal (it dominates Home + Assistant's cold start), this is not yet the answer; it would need the `schema.py` + work above first. +- **Packaging is the real ongoing cost.** Adopting this means building and shipping + per-platform wheels (cibuildwheel in CI) alongside a pure-Python sdist, and a + runtime that prefers the compiled module when present and falls back otherwise. + This is the part that amends ADR-005: the default install would carry a native + extension on supported platforms, with pure Python as the fallback rather than + the only artifact. +- **A second test lane, without coverage.** CI would run the suite against both the + pure and the compiled build, since mypyc can change behavior in edge cases (it + did, in three modules). The compiled lane runs with coverage off: mypyc code is + not traced, and the tracer can crash native code. The pure lane keeps the 100% + coverage gate. +- **The other core modules stay interpreted until reworked.** Compiling `error.py`, + `markers.py`, or `schema.py` needs real changes (a class-attribute model that + survives mypyc; a non-native `Schema` that mypyc will actually compile; an + `isinstance` path that holds for non-native subclasses), not decorators. The + `scripts/build_accelerated.py` header documents each blocker at the point it + matters. +- **The door stays open wider, not the default changed**: until this is adopted, + ADR-005 and ADR-004 stand unchanged. Adoption is a follow-up decision; this ADR + records that the engine path is validated and what it costs. + +**Revisit trigger**: adopt when the validation gain is worth the packaging and CI +cost, for example if a consumer with a tight validation loop needs it. Revisit the +wider compile (construction) only with budget to do the `schema.py` work, not as a +drop-in. The build script and numbers here are the starting point for that work. diff --git a/adr/README.md b/adr/README.md index a279eb5..7c90470 100644 --- a/adr/README.md +++ b/adr/README.md @@ -2,14 +2,15 @@ Each record captures what was chosen, the alternatives considered, and why. -| ADR | Title | -| -------------------------------------------------------------------- | ---------------------------------------------- | -| [ADR-001](001-clean-room-reimplementation-of-voluptuous.md) | Clean-room reimplementation of voluptuous | -| [ADR-002](002-mit-license.md) | MIT license | -| [ADR-003](003-astro-starlight-for-documentation.md) | Astro Starlight for documentation | -| [ADR-004](004-single-validation-engine.md) | A single validation engine (no codegen) | -| [ADR-005](005-pure-python-no-required-dependencies.md) | Pure Python, no required runtime dependencies | -| [ADR-006](006-python-support-floor.md) | Python support floor (requires-python >=3.12) | -| [ADR-007](007-self-validation-protocol.md) | A self-validation protocol for types | -| [ADR-008](008-type-to-validator-registry.md) | A type-to-validator registry for schema builders | -| [ADR-009](009-call-time-validation-context.md) | Call-time validation context | +| ADR | Title | +| ----------------------------------------------------------- | ------------------------------------------------- | +| [ADR-001](001-clean-room-reimplementation-of-voluptuous.md) | Clean-room reimplementation of voluptuous | +| [ADR-002](002-mit-license.md) | MIT license | +| [ADR-003](003-astro-starlight-for-documentation.md) | Astro Starlight for documentation | +| [ADR-004](004-single-validation-engine.md) | A single validation engine (no codegen) | +| [ADR-005](005-pure-python-no-required-dependencies.md) | Pure Python, no required runtime dependencies | +| [ADR-006](006-python-support-floor.md) | Python support floor (requires-python >=3.12) | +| [ADR-007](007-self-validation-protocol.md) | A self-validation protocol for types | +| [ADR-008](008-type-to-validator-registry.md) | A type-to-validator registry for schema builders | +| [ADR-009](009-call-time-validation-context.md) | Call-time validation context | +| [ADR-010](010-mypyc-accelerated-build.md) | mypyc as an optional accelerated build (proposed) | diff --git a/justfile b/justfile index 5acbfc5..57e3aa8 100644 --- a/justfile +++ b/justfile @@ -66,6 +66,14 @@ codspeed: uv sync --group codspeed uv run --no-sync pytest bench --codspeed --no-cov -o addopts="" +# Build the optional mypyc-accelerated engine in place (ADR-010): compiles the +# validation hot loop to a C extension next to its source. Opt-in and dev-only; +# the default build stays pure Python, so the published wheel is unaffected. Needs +# a C compiler. mypyc ships with mypy (the typing group); setuptools drives the +# build. Run `just clean` to drop the .so and return to pure Python. +build-fast: + uv run --no-sync --with setuptools python scripts/build_accelerated.py + # Run every documented Python example and verify its output comments. examples: uv run --no-sync python docs/verify_examples.py @@ -114,7 +122,8 @@ check: precommit uv run --no-sync pytest packages/pytest-probatio/tests -o addopts="" uv run --no-sync python docs/verify_examples.py -# Remove build and tooling artifacts. +# Remove build and tooling artifacts, including the opt-in accelerated .so files. clean: rm -rf dist build .pytest_cache .ruff_cache .mypy_cache .ty htmlcov .coverage coverage.xml + find src -name '*.so' -delete find . -type d -name __pycache__ -exec rm -rf {} + diff --git a/pyproject.toml b/pyproject.toml index cd7a926..6c121d2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -176,6 +176,9 @@ ignore = [ # script directory, not importable library code. They dispatch with many returns # and swallow the "expected" exceptions of the surface under test. "fuzz/**" = ["ANN", "INP001", "S101", "PLR0911", "PLR0912", "SIM105"] +# The accelerated build driver is a dev-only script (ADR-010), a script directory +# rather than importable library code. +"scripts/**" = ["INP001"] # The docs example harness is a dev-only script: it executes first-party doc # snippets (eval/exec by design), prints results, uses local imports, and runs a # branchy dispatcher over the AST. It is not importable library code. diff --git a/scripts/build_accelerated.py b/scripts/build_accelerated.py new file mode 100644 index 0000000..8aea8c4 --- /dev/null +++ b/scripts/build_accelerated.py @@ -0,0 +1,93 @@ +"""Build the optional mypyc-accelerated core in place (ADR-010). + +This is the opt-in accelerated build. It compiles the four hot core modules +(``error``, ``markers``, ``_engine``, ``schema``) into a C extension with mypyc +and leaves the rest of the package, including every validator, interpreted. The +default build stays pure Python (ADR-005); this script is something you run +yourself, never part of the published wheel. Run it with ``just build-fast``. + +The compiled ``.so`` files land next to their ``.py`` sources under ``src/`` (the +import root for the editable install), so CPython imports the native versions +when present and falls back to the pure Python sources anywhere they are absent. +Run ``just clean`` to delete them and return to pure Python. + +The validators stay interpreted on purpose: the measured win (ADR-010, 1.30x +construction and 1.38x validation on a Home-Assistant-representative workload) +already lands with just these four compiled, and keeping the validators +interpreted avoids an interpreted-inherits-compiled boundary on their shared base. +""" + +from __future__ import annotations + +import os +from pathlib import Path + +from mypyc.build import mypycify +from setuptools import setup + +# The repository root, derived from this script's location (scripts/..). +ROOT = Path(__file__).resolve().parent.parent +SRC = ROOT / "src" +# Build artifacts (generated C, object files, the staging lib) stay out of the +# tree under build/; only the final .so files land next to the sources. +BUILD = ROOT / "build" / "accelerated" + +# The validation hot loop. mypyc compiles this into one extension; everything +# else, including every validator, stays interpreted. A compiled engine that +# imports, raises, and tags interpreted classes is fine; only the reverse +# (interpreted code inheriting a compiled class) is the boundary to avoid, and +# nothing outside _engine subclasses what it defines. +# +# The other three "core" modules from the ADR-010 spike are deliberately left +# interpreted, because a compiled build of each breaks behavior the suite catches: +# +# error.py The Invalid hierarchy reads its per-subclass ``default_code`` off +# the type object (``type(self).default_code``). mypyc stores it as +# an instance slot, not a class attribute, so ``.code`` raises and the +# failure path segfaults. +# schema.py ``Schema`` must be non-native (``DataclassSchema``/``TypedDictSchema`` +# and user code subclass it, and mypyc forbids interpreted classes +# inheriting a compiled one). Compiling a non-native ``Schema`` trips an +# internal mypyc assertion on ``_compile_self``'s nested closure. +# markers.py The markers must be non-native too (they are copied and subclassed). +# A compiled non-native subclass breaks ``isinstance(x, Marker)``, so +# marker ordering and equality go wrong. +# +# These are not cosmetic adjustments; they are real blockers (see ADR-010). The +# engine is the part that compiles cleanly and keeps full behavioral parity, so it +# is the honest floor for this opt-in build. +CORE_MODULES = [str(SRC / "probatio" / "_engine.py")] + + +def main() -> None: + """Compile the core modules in place with mypyc. + + The build runs from ``src/`` so ``build_ext --inplace`` places each compiled + module next to its source (``probatio.error`` to ``src/probatio/error.so``, + the shared group module to ``src/``) and so setuptools does not read the + project's ``pyproject.toml`` (its license metadata is for the pure-Python + wheel, not this throwaway extension build). + """ + BUILD.mkdir(parents=True, exist_ok=True) + os.chdir(SRC) + + setup( + name="probatio-accelerated", + ext_modules=mypycify( + CORE_MODULES, + target_dir=str(BUILD / "c"), + group_name="probatio", + ), + script_args=[ + "build_ext", + "--inplace", + "--build-temp", + str(BUILD / "temp"), + "--build-lib", + str(BUILD / "lib"), + ], + ) + + +if __name__ == "__main__": + main()