scope.md

Scope Definition

Scope Reduction Opportunities

• Start with a Single Language Focus: One of the clearest ways to reduce scope is to limit the problem space initially to one language (or a very small number). The README uses a Rust file as an example for the proof of concept, which suggests Rust could be a candidate for MVP. By focusing on Rust ASTs only (for example), the project can avoid dealing with multi-language support, multiple parser integrations, and multiple formatters in the first iteration. Rust has a stable syntax, an excellent Tree-sitter grammar, and a canonical formatter (rustfmt), making it a good testbed. Starting with one language means the team can iron out the clean/smudge pipeline, serialization, and determinism without the complexity of switching grammars and tools per file. Once the concept is proven with Rust, adding languages one by one (maybe Python next, then JavaScript, etc.) is more manageable. Each additional language will bring its own nuances, but by then the core pipeline would be established.
• Use Existing Serialization Formats: Rather than designing a brand-new AST serialization or trying to hand-craft a binary format, the project can leverage existing formats. Tree-sitter itself can output parse trees as JSON, and many tools (like Difftastic) convert trees to S-expressions for uniform handling. For the MVP, a straightforward approach is to use JSON or a simple S-expression text as the stored format for ASTs. This has the benefit of using well-understood, debuggable text (developers can open the blob and understand it if needed) and avoids writing a complex parser/serializer for a custom binary format. Though JSON will be verbose, it's human-readable which aids development and debugging. The performance impact of JSON vs a binary format is likely secondary at this stage – Git's compression will shrink it significantly on disk. Should performance or size prove problematic, the format can be optimized later (e.g., moving to a binary representation or a condensed encoding). Also, adopting a known format might allow reuse of existing libraries; for example, if using S-expressions, there are libraries to diff and manipulate them which could be handy down the road. By scoping the serialization problem to "choose a reasonable existing format" the team saves time and reduces risk.
• Defer True Semantic Diff/Merge: It may be tempting to dive into implementing AST diff algorithms and custom merge logic, but this is an area rife with complexity. A scope-reduction recommendation is to initially rely on text diffs of formatted code (or the serialized AST) to show that the system works, and only later introduce specialized diffing. In practice, just normalizing formatting already buys a lot – many trivial diffs vanish. For the MVP, the git diff output can remain line-oriented on the pretty-printed code, which developers are used to. This means initially the tool doesn't have to supply a custom git diff driver or viewer (though it could optionally). If a semantic diff is desired for demonstration, a lightweight approach could be to integrate an existing tool like Difftastic as an external command (git ast-diff that calls Difftastic on two versions). This would showcase the potential without the team implementing it themselves from scratch. Similarly, merges in MVP can fall back to Git's standard merge on the generated code. If formatting is consistent and non-functional changes are eliminated, standard merges will already behave a bit better (no conflicts on purely moved code blocks with identical content, for example, as Git can match identical lines even if moved). By explicitly deferring the development of a full AST merge engine, the project can concentrate on the core clean/smudge pipeline first. This reduces risk – the hardest problem is left until after the basics are proven. It also avoids blocking the usefulness of the tool on having a perfect merge solution. Users could enjoy uniform formatting and cleaner diffs as initial benefits, which is still a win, while semantic merge capabilities mature separately.
• Simplify Node Identity Requirements: As noted, tracking AST node identity across versions is very hard, and implementing a robust scheme (like assigning stable IDs or content hashes to subtrees) could be a project of its own. A scope reduction here is to omit persistent node identity from the MVP. This means accepting that certain semantic moves (like moving a function) will not automatically be recognized as "the same node moved" by the tool in early versions. Instead, the focus can be on what is easier: for instance, recognizing identical subtrees via hash to assist diff might be simpler than a full identity tracking. Or just leave it to the diff algorithm to pair up similar nodes without any embedded IDs. By not attempting to embed unique IDs in the AST or carry metadata across commits initially, the team avoids a whole class of complexity (how to store the IDs, how to handle merges of IDs, etc.). This does reduce some potential functionality (like precise blame tracking of a function through moves), but those features can come later. In the interim, developers will still benefit from the other features. When the time comes to tackle node identity, the team can evaluate existing research or even consider an IR like MLIR as mentioned, but it shouldn't be in scope for the minimal viable product.
• Postpone MLIR Integration: The idea of using MLIR (Multi-Level IR) is forward-looking and could unlock advanced capabilities (rich analyses, optimizations, custom dialects for diffing). However, integrating MLIR would massively enlarge the project's scope and complexity early on. MLIR is a sophisticated compiler framework with a steep learning curve and heavy dependencies (LLVM infrastructure). Introducing it would turn a lightweight Git extension into a full compiler stack project. For the MVP, it's more pragmatic to stick with the direct AST approach using Tree-sitter. Many benefits can be delivered without MLIR in the loop. If down the road the project hits limitations that MLIR could solve (for example, needing a richer cross-language IR or more powerful transform passes), it can be revisited as an extension or v2 architecture. By explicitly postponing MLIR, the team keeps the initial implementation tractable and avoids scaring off contributors who may not be familiar with that ecosystem. In the near term, simpler standards like Tree-sitter's AST and perhaps an intermediate S-expression diff are sufficient building blocks.
• Leverage Git Features & External Tools: Rather than custom-building everything, the project can leverage Git and OS features wherever possible. For instance, to mitigate performance issues, use Git's built-in process filter support so that the filter program stays running and can parse multiple files without reloading grammars each time. This is a configuration detail, but important to include in scope to keep things efficient. Another example: if repository size bloat becomes an issue, consider compressing the AST representation as it's written (even something simple like using a binary serialization or gzip stream) – Git won't compress an already compressed blob as well, but a well-chosen representation might naturally be smaller. The key is to identify what not to write from scratch. For instance, if a need arises for a "universal AST" format, the team could examine prior art like the Babelfish UAST (Universal AST) from source{d} instead of inventing a new schema for representing every language's syntax. Using such a standard, if it fits, could save design time (though it may bring extra dependencies). Even for diffing, exploring libraries implementing GumTree or other algorithms (there are implementations in Java and perhaps Python) could allow experimentation without full commitment. By scoping the project to integration of existing components and minimal glue logic, the team can achieve more with less development effort, concentrating on the unique parts of making Git AST seamless.
• Clear MVP Goals & Stretch Goals: It's important to delineate what the Minimum Viable Product must have versus nice-to-have features. The project already lists initial POC success criteria like parsing a file, storing and regenerating it via filters, and documenting the format. Sticking to those and not expanding them is itself a form of scope control. For example, things like a fancy diff viewer, GitHub integration, or multi-language support can explicitly be labeled "future work." Keeping the MVP narrowly focused on the end-to-end pipeline for one language and demonstrating that it works in real Git operations will maximize the impact of early releases without getting bogged down. Each additional feature (even something seemingly small like handling git blame output nicely) should be weighed against the core value. Does it need to be solved now, or can we live with a rough edge initially? By constantly asking this, the team can trim the MVP to only the essentials that prove the concept and deliver immediate value (like no more fights over code formatting, and diffs that ignore whitespace). Everything else can be staged in subsequent iterations once the base is stable.

Recommended MVP Boundaries

1. Supported Language: Choose a single language for the first fully functional iteration (for example, Rust given its mention in the POC). Implement parsing and pretty-printing for that language only. This means bundling the Tree-sitter grammar for Rust and using rustfmt (or an equivalent Rust code formatter) to generate code. By limiting to one language, you ensure all edge cases and pipeline steps are handled for that language's syntax. The MVP success can be demonstrated with a repository of just Rust code where this tool is enabled. Other languages (and their specific parser/formatter) can wait until this foundation is proven.
2. End-to-End Filter Pipeline: Implement the clean filter to parse source to AST and the smudge filter to print AST back to source for the chosen language. Validate this pipeline with real Git commands. For MVP, it's acceptable that a syntax error in the code causes the commit to fail (with a clear error message) – this enforces code must parse. Document this behavior for users. Ensure that comments in the code are preserved through a round trip (e.g., a file with comments, once added and checked out again, should come back with all comments intact in the right places). This will likely involve using the CST from Tree-sitter, which includes comment tokens, and making sure the pretty-printer doesn't drop them. Achieving lossless conversion (aside from whitespace formatting) for one representative language is a critical MVP milestone.
3. Deterministic AST Serialization: Pick and implement a serialization format for the AST that is used as the Git blob content. For simplicity, this could be JSON or a line-based S-expression notation. The format should be deterministic (no extraneous ordering issues or IDs that change on each parse). For instance, if using JSON, consistently order object keys or use arrays such that the same code always yields byte-for-byte the same AST representation. Document this format in the repository so that contributors know what the stored data looks like. It's fine if the format is verbose at first – clarity and correctness trump optimization for MVP. The key is that given the same source file, everyone's environment produces the exact same serialized AST (to avoid phantom diffs due to environment differences).
4. Integration with Git (Configuration): Provide an easy way to set up the filters. For MVP, a script or manual instructions can guide adding entries to .gitattributes for the target language (e.g., *.rs filter=git-ast) and configuring Git to use the git-ast filter driver. The project could supply a sample .gitattributes and the corresponding Git config snippet. The goal is that a user following the README can enable Git AST on a sample repository with minimal effort. This includes installing the git-ast tool (which might be a binary or script) and verifying that git add and git checkout automatically invoke it. Ensuring this setup is smooth is part of MVP – it doesn't matter how good the tool is if users can't figure out how to turn it on. Thus, clear installation steps (possibly even an install.sh or a Homebrew/Rust crate if applicable) should be in scope for the MVP.
5. Basic Diff and Merge Behavior: For the MVP, formal semantic diff/merge features can be very limited. It's sufficient that git diff on the working directory shows meaningful results. Since the working copy is formatted code, git diff will naturally show changes in code terms (ignoring any formatting differences because those never reach the repo). It's worth testing scenarios like reordering functions or renaming a variable purely cosmetically – ideally these should result in minimal diff output. If the team wants, they can implement a custom diff driver that pretty-prints the ASTs of two commits and then diffs them, but this can be optional. What's more important is to document how diff and merge behave under the system. For merges, initially rely on Git's normal text merge on the pretty-printed files. The MVP should note that complex refactor merges may not be auto-resolved yet (that's future work), but confirm that no worse conflicts occur due to the tool. In other words, ensure that storing ASTs doesn't break Git's ability to merge (by testing merging two branches that both edit different parts of a file, etc.). The success criterion here is that merging with the filters enabled yields the same result as merging the formatted code would – which should be the case if both sides produce their ASTs and then smudge to code for the merge. Any corner case where this fails should be identified and possibly disallowed in MVP.
6. Performance Baseline Testing: As part of MVP validation, test the filter on a moderately sized project (maybe the target language's own standard library or a popular open source project in that language). Measure the time it takes to add files and checkout branches with the filter on. If performance is acceptable (say, adding a file introduces only a small delay, and checking out 100 files doesn't take orders of magnitude longer than normal), that's good enough for MVP. If any one step is painfully slow, consider quick optimizations: for example, enabling the filter process mode in Git config so that the parser stays resident. It would be beneficial for the MVP to include notes or config for this "performance mode." The idea is not to fully optimize now, but to ensure there are no show-stoppers. Memory usage should also be observed (a large AST can consume a lot of RAM); the MVP should at least flag if certain large files cause issues. Setting an upper bound (e.g., files over a certain size bypass the filter) might be a contingency to include if needed, to avoid pathological cases. The end result should be a reasonable confidence that the approach scales to real-world file sizes, at least for the chosen language.
7. Documentation & Developer Guidance: A successful MVP isn't just code – it includes documentation explaining the new workflow and its limitations. The project should ship a README (or user guide) that describes how the clean/smudge process works in practice, what developers need to know (for example, "don't edit the .ast files in the git object store – always edit the working copy code!" or "if your commit is rejected due to parse errors, fix the syntax or use bypass mode X"). It should also mention known limitations (e.g., "currently only supports Rust, and merges are still text-based") so expectations are managed. Providing a simple example in the docs can be very powerful: for instance, show a small code change and how the diff looks with Git AST versus without. This will help others appreciate the value. Additionally, guidelines on how to temporarily disable the filter (in case something goes wrong and a user needs to get their code in) could be included for safety. From a project leadership perspective, delivering a well-documented MVP will make it much easier to attract users and contributors for the next phases.