Migrate tree-sitter support to MODULE.bazel (carbon-language#4783)

The WORKSPACE file is deprecated; support is already off by default, and
it'll be removed in the next major bazel release. Our main dependency is
tree-sitter, and I'm trying to address that here.

We're currently using https://github.com/elliottt/rules_tree_sitter, but
that hasn't been updated in a couple years, meaning it lacks
MODULE.bazel support. In the registry, there's
https://registry.bazel.build/modules/tree-sitter-bazel, but this is only
the *parser* libraries of tree-sitter, not the *generator*. I'm using it
for that much, at least.

For the *generator*, which transforms grammar.js to parser.c/h, I'm just
requiring a non-hermetic invocation (i.e., people who want to work on it
will need to install tree-sitter; see the README.md updates). I tried
running it manually, but parser.c is about 600 KB; pre-commit rejects
files that large and I don't think an exception makes sense to override
for this (it'd probably also grow substantially if the grammar were
updated to cover more syntax). In order to make the non-hermetic call
not break "bazel build //..." for most developers, I'm marking most
targets in the package as manual.

Note, I did look long and hard at using `aspect_rules_js`/`rules_nodejs`
to invoke npm. This took a lot of time, and I have a commit that's
mostly working, except I hit a point where it uses `declare_symlink`
which we disallow for compatibility reasons (commit "Lots of work for
figuring out rule_js uses declare_symlink" on the PR). As a consequence,
I think we can't use the primary supported ways to have hermetic npm
calls.

Also, `treesitter` -> `tree_sitter` because it's generally called
`tree-sitter`, two words. We even had a `treesitter/src/tree_sitter`
directory so it's a bit inconsistent.

As far as bugs here, the parser library breaks bazel queries, e.g. the
error:
```
ERROR: Evaluation of query "somepath(//..., @llvm-project//third-party/unittest:gtest)" failed: preloading transitive closure failed: no such package '@@[unknown repo 'platforms' requested from @@tree-sitter-bazel+]//': The repository '@@[unknown repo 'platforms' requested from @@tree-sitter-bazel+]' could not be resolved: No repository visible as '@platforms' from repository '@@tree-sitter-bazel+'
```

I'm just excluding tree_sitter from queries where I can to work around
the error.

Author: jonmeow

.bazelrc

@@ -57,11 +57,6 @@ build --action_env=CMAKE_SYSROOT --host_action_env=CMAKE_SYSROOT
 build --per_file_copt=external/.*\.(c|cc|cpp|cxx)$@-w
 build --host_per_file_copt=external/.*\.(c|cc|cpp|cxx)$@-w
 
-# The `rules_treesitter` synthesized libraries don't allow us to inject flags,
-# and compile generated code where we can't fix warnings.
-build --per_file_copt=utils/treesitter/_treesitter.tree_sitter/.*\.c$@-w
-build --host_per_file_copt=utils/treesitter/_treesitter.tree_sitter/.*\.c$@-w
-
 # Default dynamic linking to off. While this can help build performance in some
 # edge cases with very large linked executables and a slow linker, between using
 # fast linkers on all platforms (LLD and the Apple linker), as well as having
@@ -128,17 +123,6 @@ build --allow_unresolved_symlinks=false
 # RC file here if present.
 try-import %workspace%/user.bazelrc
 
-# Incompatible with `rules_tree_sitter`.
-# TODO: WORKSPACE will be removed in bazel 9, and we need to move off.
-# TODO: The registry has a different treesitter rule set, and we should
-# investigate switching. See:
-# https://registry.bazel.build/modules/tree-sitter-bazel
-common --enable_workspace
-# This is on by default in bazel 8.
-common --incompatible_disallow_empty_glob=false
-# common --incompatible_auto_exec_groups
-# common --incompatible_disable_starlark_host_transitions
-
 # This excludes things like rules_android to reduce warnings.
 # TODO: There's a pending fix, so hopefully we can remove it soon.
 # - Issue: https://github.com/bazelbuild/bazel/issues/23929
@@ -150,11 +134,19 @@ common --incompatible_autoload_externally=+@rules_java,+@rules_python,+@rules_sh
 # TODO: Enable the flag once compatibility issues are fixed.
 # common --incompatible_disable_non_executable_java_binary
 
+# Incompatible with the clang-tidy build mode.
+# TODO: Enable the flag once compatibility issues are fixed.
+# common --incompatible_auto_exec_groups
+
 # Incompatible with `rules_cc`.
 # TODO: Enable the flag once compatibility issues are fixed.
 # common --incompatible_no_rule_outputs_param
 # common --incompatible_stop_exporting_language_modules
 
+# Incompatible with `rules_flex`.
+# TODO: Enable the flag once compatibility issues are fixed.
+# common --incompatible_disable_starlark_host_transitions
+
 # Incompatible with `rules_pkg`.
 # TODO: Enable the flag once compatibility issues are fixed.
 # common --incompatible_disable_target_default_provider_fields

.gitignore

@@ -35,7 +35,7 @@
 # vim temporary files
 .*.sw[a-p]
 
-# generated by utils/treesitter/helix.sh
+# generated by utils/tree_sitter/helix.sh
 /.helix/
 
 # Ignore .DS_Store files

MODULE.bazel

@@ -28,12 +28,13 @@ http_archive = use_repo_rule(
     "http_archive",
 )
 
-bazel_dep(name = "bazel_skylib", version = "1.7.1")
-bazel_dep(name = "rules_pkg", version = "1.0.1")
 bazel_dep(name = "abseil-cpp", version = "20240722.0.bcr.2")
-bazel_dep(name = "re2", version = "2024-07-02.bcr.1")
-bazel_dep(name = "googletest", version = "1.15.2")
+bazel_dep(name = "bazel_skylib", version = "1.7.1")
 bazel_dep(name = "google_benchmark", version = "1.8.5")
+bazel_dep(name = "googletest", version = "1.15.2")
+bazel_dep(name = "re2", version = "2024-07-02.bcr.1")
+bazel_dep(name = "rules_pkg", version = "1.0.1")
+bazel_dep(name = "tree-sitter-bazel", version = "0.24.4")
 
 # The registry only has an old version. We use that here to avoid a miss but
 # override it with a newer version.

MODULE.bazel.lock

@@ -18,14 +18,17 @@ filegroup(
         "//migrate_cpp:rewriter",
         "//migrate_cpp/cpp_refactoring",
         "//toolchain/install:carbon-busybox",
-        "//utils/treesitter",
+        # The tree sitter rules can't be queried; evaluation fails on
+        # @platforms.
+        # "//utils/tree_sitter",
     ],
     tags = ["manual"],
 )
 
 genquery(
     name = "non_test_cc_deps.txt",
-    expression = "kind('cc.* rule', deps(//bazel/check_deps:non_test_cc_rules))",
+    expression =
+        "kind('cc.* rule', deps(//bazel/check_deps:non_test_cc_rules))",
     opts = [
         "--notool_deps",
         "--noimplicit_deps",

WORKSPACE

@@ -38,7 +38,7 @@ in Unicode Normalization Form C (NFC).
 <!--
 Keep in sync:
 - utils/textmate/Syntaxes/Carbon.plist
-- utils/treesitter/queries/highlights.scm
+- utils/tree_sitter/queries/highlights.scm
 -->
 
 The following words are interpreted as keywords:

bazel/check_deps/BUILD

@@ -95,12 +95,12 @@ file_test(
 )
 
 filegroup(
-    name = "treesitter_testdata",
+    name = "tree_sitter_testdata",
     srcs = glob(
         ["testdata/**/*.carbon"],
         exclude = [
             "testdata/**/fail_*",
         ],
     ),
-    visibility = ["//utils/treesitter:__pkg__"],
+    visibility = ["//utils/tree_sitter:__pkg__"],
 )

docs/design/lexical_conventions/words.md

@@ -36,7 +36,9 @@ def _build_generated_files(bazel: str) -> None:
     kinds_query = (
         "filter("
         ' ".*\\.(h|cpp|cc|c|cxx|def|inc)$",'
-        ' kind("generated file", deps(//...))'
+        # tree_sitter is excluded here because it causes the query to failure on
+        # `@platforms`.
+        ' kind("generated file", deps(//... except //utils/tree_sitter/...))'
         ")"
     )
     generated_file_labels = subprocess.check_output(

explorer/BUILD

@@ -38,7 +38,11 @@ def main() -> None:
     args = [
         scripts_utils.locate_bazel(),
         "query",
-        "somepath(//..., @llvm-project//third-party/unittest:gtest)",
+        "somepath("
+        # tree_sitter is excluded here because it causes the query to failure on
+        # `@platforms`.
+        + "//... except //utils/tree_sitter/..., "
+        + "@llvm-project//third-party/unittest:gtest)",
     ]
     p = subprocess.run(
         args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding="utf-8"

scripts/create_compdb.py

@@ -22,4 +22,4 @@ developers and developers writing Carbon code.
 Any editor that supports Language server protocol and/or tree-sitter is
 supported. The editor just needs to be configured manually.
 `bazel build //toolchain` produces the language server binary.
-`utils/treesitter` contains the treesitter grammar.
+`utils/tree_sitter` contains the tree-sitter grammar.

scripts/forbid_llvm_googletest.py

@@ -11,15 +11,15 @@ ROOT="$(git rev-parse --show-toplevel)"
 mkdir -p ~/.config/nvim/{lua,parser,queries}
 
 # add highlight queries
-ln -sTf "$PWD/utils/treesitter/queries" ~/.config/nvim/queries/carbon
+ln -sTf "$PWD/utils/tree_sitter/queries" ~/.config/nvim/queries/carbon
 
 # add carbon.lua
 ln -sf "$PWD/utils/nvim/carbon.lua" ~/.config/nvim/lua/carbon.lua
 
 # load carbon.lua on startup
 grep 'require "carbon"' ~/.config/nvim/init.lua || echo 'require "carbon"' >> ~/.config/nvim/init.lua
 
-# build treesitter
-cd utils/treesitter
+# build tree_sitter
+cd utils/tree_sitter
 tree-sitter generate
 clang -o ~/.config/nvim/parser/carbon.so -shared src/parser.c src/scanner.c -I ./src -Os -fPIC

utils/README.md

@@ -0,0 +1,118 @@
+# Part of the Carbon Language project, under the Apache License v2.0 with LLVM
+# Exceptions. See /LICENSE for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library", "cc_test")
+
+package(default_visibility = ["//bazel/check_deps:__pkg__"])
+
+# See README.md for instructions on tree-sitter setup and use. These rules are
+# manual because the tree-sitter invocation is non-hermetic, and most developers
+# won't have it installed; we don't want to break "bazel test //..." if we can
+# avoid it.
+#
+# We use tree-sitter non-hermetically for two key reasons:
+#
+# - The main way of hermetically using npms in bazel, `aspect_rules_js`, uses
+#   declare_symlink; we disallow that for important compatibility reasons.
+# - When generated, src/parser.c is over 500 KB, which is larger than we want to
+#   check in. It should also be expected to grow if the grammar becomes more complete.
+
+# Convenience target for running all tests, including manual tests.
+test_suite(
+    name = "tests",
+    tags = ["manual"],
+    tests = [
+        ":explorer_tests",
+        ":string_fail_tests",
+        ":string_tests",
+    ],
+)
+
+# Call tree-sitter to generate parser files.
+genrule(
+    name = "parser_files",
+    srcs = ["grammar.js"],
+    outs = [
+        "src/parser.c",
+        "src/tree_sitter/parser.h",
+    ],
+    cmd = "tree-sitter generate $(location grammar.js) &&\n" +
+          "cp src/parser.c $(location src/parser.c) &&\n" +
+          "cp src/tree_sitter/parser.h $(location src/tree_sitter/parser.h)",
+    tags = ["manual"],
+)
+
+cc_library(
+    name = "parser",
+    srcs = [
+        "src/scanner.c",
+        ":src/parser.c",
+    ],
+    hdrs = [":src/tree_sitter/parser.h"],
+    copts = ["-Wno-missing-prototypes"],
+    tags = ["manual"],
+    deps = ["@tree-sitter-bazel//:tree-sitter"],
+)
+
+cc_binary(
+    name = "test_runner",
+    testonly = 1,
+    srcs = ["test_runner.cpp"],
+    tags = ["manual"],
+    deps = [
+        ":parser",
+    ],
+)
+
+cc_test(
+    name = "explorer_tests",
+    size = "small",
+    srcs = ["test_runner.cpp"],
+    args = ["$(locations //explorer:tree_sitter_testdata)"],
+    data = ["//explorer:tree_sitter_testdata"],
+    tags = ["manual"],
+    deps = [
+        ":parser",
+    ],
+)
+
+filegroup(
+    name = "string_testdata",
+    srcs = glob(
+        ["testdata/string/*.carbon"],
+        exclude = ["testdata/string/fail_*.carbon"],
+    ),
+)
+
+filegroup(
+    name = "string_fail_testdata",
+    srcs = glob(["testdata/string/fail_*.carbon"]),
+)
+
+cc_test(
+    name = "string_tests",
+    size = "small",
+    srcs = ["test_runner.cpp"],
+    args = ["$(locations :string_testdata)"],
+    data = [":string_testdata"],
+    tags = ["manual"],
+    deps = [
+        ":parser",
+    ],
+)
+
+cc_test(
+    name = "string_fail_tests",
+    size = "small",
+    srcs = ["test_runner.cpp"],
+    args = ["$(locations :string_fail_testdata)"],
+    data = [":string_fail_testdata"],
+    env = {
+        "FAIL_TESTS": "1",
+    },
+    tags = ["manual"],
+    deps = [
+        ":parser",
+    ],
+)

utils/nvim/setup.sh

@@ -1,13 +1,22 @@
+# Tree-sitter grammar for Carbon
+
 <!--
 Part of the Carbon Language project, under the Apache License v2.0 with LLVM
 Exceptions. See /LICENSE for license information.
 SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 -->
 
-# Tree-sitter grammar for Carbon
-
 Tree-sitter is currently used for syntax highlighting in supported editors.
 
+## Development
+
+We use a non-hermetic tree-sitter invocation, so it must be installed locally.
+To install tree-sitter, run:
+
+```
+npm install -g tree-sitter-cli
+```
+
 ## Editor Installation
 
 ### Helix

utils/treesitter/.gitignore utils/tree_sitter/.gitignore

@@ -7,7 +7,7 @@
 set -euo pipefail
 
 ROOT="$(git rev-parse --show-toplevel)"
-cd "$ROOT/utils/treesitter"
+cd "$ROOT/utils/tree_sitter"
 
 tree-sitter generate --no-bindings
 

utils/tree_sitter/BUILD

@@ -2,7 +2,7 @@
 // Exceptions. See /LICENSE for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
-#include "tree_sitter/parser.h"
+#include "utils/tree_sitter/src/tree_sitter/parser.h"
 
 enum TokenType {
   BINARY_STAR,

utils/treesitter/README.md utils/tree_sitter/README.md

@@ -3,7 +3,6 @@
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 #include <tree_sitter/api.h>
-#include <tree_sitter/parser.h>
 
 #include <cstdlib>
 #include <filesystem>
@@ -13,6 +12,8 @@
 #include <string>
 #include <vector>
 
+#include "utils/tree_sitter/src/tree_sitter/parser.h"
+
 extern "C" {
 auto tree_sitter_carbon() -> TSLanguage*;
 }
@@ -29,7 +30,7 @@ static auto ReadFile(std::filesystem::path path) -> std::string {
 // TODO: use file_test.cpp
 auto main(int argc, char** argv) -> int {
   if (argc < 2) {
-    std::cerr << "Usage: treesitter_carbon_tester <file>...\n";
+    std::cerr << "Usage: test_runner <file>...\n";
     return 2;
   }