feat(runfiles): support for --incompatible_compact_repo_mapping_manifest

[jklukas]

Under bzlmod, the repo mapping can become quite large (i.e. tens of megabytes)
because its size scales as a factor of the number of repos in the transitive
dependencies.

To address this, the --incompatible_compact_repo_mapping_manifest flag was introduced.
This changes the repo mapping formation to use prefixes (instead of exact repo names)
for mapping things.

To make this work with the runfiles library, the code has to be updated to handle
these prefixes instead of just exact strings.

Fixes #3022.

[gemini-code-assist]

Summary of Changes

Hello @jklukas, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates support for Bazel's --incompatible_compact_repo_mapping_manifest flag into the Python runfiles library. This is a critical enhancement for Bazel users leveraging bzlmod, as it addresses the issue of excessively large repository mapping manifests by introducing prefix-based mapping resolution. The change ensures that the runfiles library can correctly locate files even when the repository mapping is in its compact, memory-optimized format, thereby improving the efficiency and scalability of Bazel builds with extensive dependency trees.

Highlights

• Compact Repository Mapping Support: The Python runfiles library now supports Bazel's --incompatible_compact_repo_mapping_manifest flag.
• Memory Optimization: This feature enables prefix-based repository mappings, significantly reducing the size of the repo mapping manifest (from tens of megabytes to much less) for large bzlmod dependency graphs.
• New _RepositoryMapping Class: A new internal class _RepositoryMapping was introduced to intelligently parse and resolve both exact and prefix-based repository mappings, ensuring correct runfiles resolution.
• Enhanced Rlocation Method: The Rlocation method in the Runfiles class has been updated to utilize the new _RepositoryMapping logic, handling the complexities of prefix-based lookups with proper precedence.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[jklukas]

cc @fmeum for review

[gemini-code-assist]

Code Review

This pull request adds support for Bazel's --incompatible_compact_repo_mapping_manifest flag, which is a great feature for improving performance with bzlmod. The implementation is solid and includes a comprehensive set of tests. My review includes suggestions for a performance optimization in the new mapping logic, a fix for an outdated docstring, and some refactoring opportunities in both the implementation and the tests to improve maintainability and readability.

[rickeylev]

I was going to push some changes, but noticed that "maintainer can modify" isn't set.

Mostly LGTM, but please remove all the Mapping API compatibility code. As an internal class, there's no need for it to support the wide range of dict operations.

[jklukas]

Thank you for the quick review! I believe all comments are now addressed. I removed the Mapping compat; now we just have a lookup method and an is_empty method as needed for the rest of the logic.

[jklukas]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for Bazel's --incompatible_compact_repo_mapping_manifest flag, which uses prefix-based repository mappings. The implementation introduces a _RepositoryMapping class to handle both exact and new prefix-based mappings, which is a clean way to encapsulate the logic. The accompanying tests are thorough and cover various scenarios including precedence and ordering.

I've found a couple of areas for improvement. First, in Rlocation, there's an opportunity to refactor the logic to avoid a redundant lookup and remove some unreachable code, making it more efficient and maintainable. Second, in _RepositoryMapping.create_from_file, adding a check for empty lines in the manifest file would make the parsing more robust.

Overall, this is a great addition that addresses an important performance issue with large dependency graphs in Bazel.

[jklukas]

Gentle ping for reviewers to take another look after pushing fixes.

[rickeylev]

Thanks for implementing this!

[jklukas]

Thanks for the review and getting this landed!