feat: Add prebuilt tree-sitter grammars for Semble - #1
Merged
Merged
Conversation
…lti-platform wheels Replaces the template placeholders with a working package: a manifest-driven tree-sitter loader (get_language/get_parser) that extracts a bundled, platform-specific grammar archive into a local cache with no runtime network access. tools/build_grammars.py compiles 45 grammars from pinned upstream commits (with license/provenance tracking) for macOS, Linux x86_64/arm64, and Windows (cross-compiled via mingw-w64); tools/build_wheels.py packages one platform-tagged wheel per architecture. CI now builds grammars before testing, and the release workflow builds real per-platform wheels. Compiled grammar archives are intentionally not committed (see .gitignore): builds aren't byte-reproducible, so tracking them would bloat history with a fresh multi-MB binary diff on every language addition. They're rebuilt by CI and by tools/build_grammars.py on demand.
… warnings - tests/test_loader.py: cache-reuse test hardcoded .dylib, which would fail on Linux (.so) and Windows (.dll) CI runners; derive the filename from the platform manifest instead. - pyproject.toml: tree-sitter>=0.21,<1.0 allowed versions whose Language() API is incompatible with the loader's single-arg PyCapsule call (verified 0.21.3-0.23.0 fail, 0.23.1+ works); tightened to >=0.23.1,<0.27. - pyproject.toml: package-data glob referenced _grammars/ and _grammars/licenses/ without declaring them as included packages, which setuptools warned about on every wheel build; fixed via packages.find with namespaces=true. - setup.py: import setuptools' own integrated bdist_wheel command instead of the deprecated wheel.bdist_wheel; bump the setuptools floor to 70.1. - Makefile: add missing lint/typecheck targets (make lint was failing). - tools/: drop file-level module docstrings and "# --- batch N ---" section separators per house style. - release.yaml: smoke-test each built wheel (tools/smoke_test.py) before publishing, instead of publishing straight after build.
- semble_grammars/ -> src/semble_grammars/: standard src-layout, avoids accidentally importing from the working directory instead of the installed package. - _cache.py -> cache.py, _loader.py -> loader.py, _platform.py -> platform.py: drop the leading-underscore "private module" convention in favor of plain names. platform.py's own `import platform` still resolves to the stdlib module (Python 3 imports are absolute-only), verified working. - Updated all imports, tool script paths (tools/build_grammars.py, tools/build_wheels.py), pyproject.toml packaging config (package-dir/packages.find), Makefile, pre-commit mypy args, .gitignore, and README references accordingly.
…ader Language additions (45 -> 56), per repo-frequency-vs-build-cost review: sql, csharp, jinja2, json5, powershell, scss, astro, starlark, gotmpl, rst, groovy, plus terraform as an alias for hcl (same syntax, no separate grammar). GrammarSpec gained symbol_override (csharp's compiled symbol is tree_sitter_c_sharp, not tree_sitter_csharp) and generate=True, which runs `tree-sitter generate` for grammars whose repo doesn't commit a pre-built parser.c (needed for sql). Docker cross-build image moved from debian:bookworm-slim to debian:trixie-slim: bookworm's glibc (2.36) is too old for the tree-sitter CLI's prebuilt binary (needs 2.39+). Verified this doesn't affect the manylinux2014 wheel tag's promise - the compiled grammars themselves only require up to GLIBC_2.14, checked via objdump -T on the built .so files. loader.py: keep every dlopen'd library's CDLL handle alive in a module-level dict alongside the Language built from it, and raise a clear GrammarLoadError instead of proceeding with a null/invalid pointer on a missing symbol or failed load. ctypes.CDLL has no __del__ and empirically doesn't unload on GC, so this wasn't an active bug, but it's cheap to stop relying on that being ctypes' behavior forever.
…mpile The first real CI run (PR #1) failed all 5 windows-latest jobs: clang: error: unsupported option '-fPIC' for target 'x86_64-pc-windows-msvc' Plain `clang` on Windows targets MSVC by default, which rejects the GNU-style flags this build relies on and doesn't export symbols the way ctypes expects without extra __declspec annotations we can't add to third-party grammar sources. Switched build_native() to use MinGW-w64 gcc on Windows specifically - the same compiler already proven to work via build_windows_cross()'s cross-compilation path - and added a `choco install mingw` step to both ci.yaml and release.yaml's Windows jobs. macOS/Linux builds are unaffected (still plain clang).
…rgv[0] Second CI failure after the mingw fix: all 45 pre-existing grammars now compiled fine on windows-latest, but sql's `tree-sitter generate` step hit FileNotFoundError (WinError 2). npm's global install puts a .cmd shim on Windows, and subprocess/CreateProcess won't resolve that extension from a bare "tree-sitter" argv[0] the way a shell would. Use shutil.which() to resolve the actual executable path, same pattern already used for the pre-flight existence check.
…indows rename race
macos-latest/3.10: tree-sitter generate hit the 60s CLONE_TIMEOUT_SECONDS
(reused from the git-fetch timeout) under runner load. Gave it its own
longer budget (GENERATE_TIMEOUT_SECONDS = 180) instead of sharing one meant
for a shallow git fetch.
windows-latest/3.11 and /3.14: test_extract_atomic_survives_concurrent_first_use
failed with PermissionError('Access is denied') from os.replace. Unlike
POSIX, Windows enforces mandatory file locking, so multiple threads racing
os.replace(tmp, dest_path) onto the same destination can transiently deny
all but the winner. windows-latest/3.10 and /3.12 passed in the same run,
confirming this is a race, not a broken build. extract_atomic now treats a
failed replace as non-fatal if dest_path exists afterward - some other
caller's replace already produced an equally valid file.
windows-latest/3.10 and /3.12 already passing in this same run confirms the
mingw compiler fix and tree-sitter CLI path fix (previous two commits) are
both correct - these are new, narrower issues the first clean build exposed.
Pringled
changed the title
…RM64 builds get_parser() enforces tree-sitter's ABI compatibility check, which get_language() defers; 25 of 56 bundled grammars are ABI 15 and fail under tree-sitter<0.25 with "Incompatible Language version". Raise the lower bound accordingly and switch the all-language test to get_parser() so this regresses loudly. Also pin tree-sitter-cli to a known-good version across all build/CI paths for reproducible codegen, and register QEMU before the Linux release job's ARM64 Docker cross-build (previously unregistered, likely to fail on GitHub-hosted x86_64 runners).
README and the missing-CLI error message still told developers to install tree-sitter-cli unpinned; align with the CI/release pins.
stephantul
self-requested a review
July 20, 2026 08:39
Confidence Score: 4/5The remaining Linux compatibility issue should be fixed before merging.
scripts/build_grammars.py and .github/workflows/release.yaml Reviews (2): Last reviewed commit: "Fix release platform compatibility [skip..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Fills gaps in mainstream config/prose/language coverage (70 -> 75 bundled grammars). Also accept LICENCE (British spelling) when collecting upstream license files, needed by one of the new sources.
Fix platform list ordering and drop the redundant Development section.
ci.yaml previously only fully built+tested linux-x86_64 and windows-x86_64, with macos-arm64 build-only and path-filtered to grammar-touching PRs. linux-arm64, windows-arm64, and macos-x86_64 had zero coverage until the release workflow itself ran. Since the release workflow's build jobs mirror what's added here, extend the same patterns into ci.yaml so every platform is exercised on every push, not just at release time. Folds macos-build.yaml's job into ci.yaml (now unconditional and tested) and removes the now-redundant file.
- Regenerate THIRD_PARTY_NOTICES.md: it had gone stale (72/75 rows, missing fortran/objc/perl) after an earlier local experiment rebuilt-then-reverted sources.json without re-syncing this generated file. - Track uv.lock instead of ignoring it, so the dev/CI toolchain resolves reproducibly. Unpinned dev deps let `uv sync` pick up an incompatible pydoclint/docstring_parser_fork pairing (or a corrupted cache entry) that broke `make lint` locally while the isolated pre-commit hook, pinned separately, stayed green. - Extend the release smoke test to load every bundled language via get_language(), not just python. Verified empirically: all 75 linux-x86_64 libraries built in debian:trixie-slim actually load fine under a real manylinux2014 (glibc 2.17) container today, but nothing was checking that beyond the python grammar, so a future regression could ship undetected.
stephantul
approved these changes
Jul 31, 2026
windows-arm64 was silently building x86_64 DLLs (the mingw gcc fallback is x86_64-only), which then failed to load into the ARM64 Python interpreter with WinError 193. Use cl via ilammy/msvc-dev-cmd for both Windows architectures, with an explicit VSCMD_ARG_TGT_ARCH preflight check so a target/environment mismatch fails fast instead of producing a wrong-arch binary.
QEMU-emulated ARM64 clang compiling generated parser C files at -O2 is extremely slow and timed out tree-sitter generate for the sql grammar. ubuntu-24.04-arm gives native ARM64 execution for both the grammar build and the manylinux2014_aarch64 smoke test, so the docker --platform flags now match the host instead of triggering emulation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR bundles 56 pinned tree-sitter grammars for macOS, Linux, and Windows so Semble can parse files without runtime downloads. It also:
Tested in Semble (MinishLab/semble@main...grammars) and it produces the exact same chunks on our benchmarks (while being a bit faster as well).