Repository navigation
Support a shared directory for compiled lint libraries - #2179
Conversation
e8031c4 to
3bc93fa
Compare
|
@dblock Thanks very much for working on this.
Why the |
A fork or a separate local checkout that hasn't been released. Importantly, --no-build could silently load the wrong revision. The actual layout is /-/. For Git dependencies, the hash identifies the repository, resolved commit, and package path—not the consuming workspace—so different consumer worktrees still share the cache. |
1660c8e to
dd0ca1d
Compare
If I understand correctly, this is a problem with how As such, I would rather disentangle the fix from this PR and #2180. That would also simplify the logic because shared Does that make sense? Also, is the collision problem something you are running into? Or were you just trying to be proactive? |
|
Yes, that makes sense. The problem I’m actually encountering is repeated compilation across consumer worktrees; collision prevention was proactive. Would you prefer that I drop source partitioning from both PRs for now, preserve the existing cache layout and behavior, and leave collision handling for separate work? That should also let us remove the So the plan would be to
|
Yes.
If it's not a huge ask, could we keep this PR and update #2180 once this one is settled? I actually find them easier to review as separate PRs. |
dd0ca1d to
d229697
Compare
Whatever you need to get the changes merged, don't hold back asking! This is done, I updated this PR with what we discussed (I hope). Thanks for taking the time for the review. |
|
Sorry. Rereading our discussion, I think I may have been unclear. I was hoping that this PR could be what it was before, just without collision handling. So, essentially, the addition of the Would that be possible? |
d229697 to
651e25f
Compare
Updated. The consolidation PR will remove the setting ;) |
smoelius
left a comment
There was a problem hiding this comment.
Please let me know if I have misunderstood your changes in any way.
| DYLINT_LIBRARIES_DIR (default: <workspace target>/dylint/libraries) is an absolute path where | ||
| Dylint builds lint libraries, using <directory>/<toolchain>/release. Set it to share builds | ||
| across projects and worktrees. It does not isolate libraries by source. |
There was a problem hiding this comment.
| DYLINT_LIBRARIES_DIR (default: <workspace target>/dylint/libraries) is an absolute path where | |
| Dylint builds lint libraries, using <directory>/<toolchain>/release. Set it to share builds | |
| across projects and worktrees. It does not isolate libraries by source. | |
| DYLINT_LIBRARIES_DIR (default: <target directory>/dylint/libraries) is an absolute path where | |
| Dylint builds lint libraries, using <directory>/<toolchain>/release. Set it to share builds | |
| across projects. |
These are nits. I think "target directory" is more clear than "workspace target"; "worktrees" is a Git concept, and IMO tangential; and the last sentence anticipates a bug fix and will just get removed anyway.
|
|
||
| Dylint downloads and builds each entry, similar to how Cargo downloads and builds a dependency. The resulting `target/release` directories are searched for files with names of the form that Dylint recognizes (see [Library requirements] under [How Dylint works]). | ||
|
|
||
| By default, these builds use `<workspace target>/dylint/libraries/<toolchain>/release`. Set `DYLINT_LIBRARIES_DIR` to an absolute path to reuse builds across projects or worktrees under `<directory>/<toolchain>/release` instead; the directory is created as needed. Same-name libraries using one toolchain can replace each other, and `--no-build` does not validate their source. |
There was a problem hiding this comment.
| By default, these builds use `<workspace target>/dylint/libraries/<toolchain>/release`. Set `DYLINT_LIBRARIES_DIR` to an absolute path to reuse builds across projects or worktrees under `<directory>/<toolchain>/release` instead; the directory is created as needed. Same-name libraries using one toolchain can replace each other, and `--no-build` does not validate their source. | |
| By default, these builds use `<target directory>/dylint/libraries/<toolchain>/release`. Set `DYLINT_LIBRARIES_DIR` to an absolute path to reuse builds under `<directory>/<toolchain>/release` instead. The directory is created as needed. |
For reasons similar to the help message. Also, "projects...under <directory>/<toolchain>/release" sounds like that is where the projects live.
| let cache = tempdir().unwrap(); | ||
| let libraries_directory = cache.path().join("shared libraries"); |
There was a problem hiding this comment.
| let cache = tempdir().unwrap(); | |
| let libraries_directory = cache.path().join("shared libraries"); | |
| let libraries_directory = tempdir().unwrap(); |
| .unwrap(); | ||
| let target_directory = fixture.path().join("target"); | ||
|
|
||
| let command = || { |
There was a problem hiding this comment.
| let command = || { |
I'm recommending the --no-build part of this test go away; hence, command should be usable directly.
| command | ||
| }; |
There was a problem hiding this comment.
| command | |
| }; |
See above.
| let release_directory = libraries_directory.join(&toolchain).join("release"); | ||
| assert!( | ||
| release_directory | ||
| .join(library_plain_filename("collapsible_unwrap")) | ||
| .is_file() | ||
| ); |
There was a problem hiding this comment.
Please move the release_directory check to after the copied_library_directory check. IMO, that overall order is more clear, i.e., the target directory is checked first, then where the library is copied to, then where the library ultimately lands.
Please leave a newline after this let and assert! are moved. Please also separate them from the copied_library_directory check with a newline.
| command() | ||
| .args(["dylint", "--all", "--no-build"]) | ||
| .assert() | ||
| .success() | ||
| .stderr( | ||
| predicate::str::contains("\nwarning: `unwrap`s that could be combined\n") | ||
| .and(predicate::str::contains("Compiling").not()), | ||
| ); |
There was a problem hiding this comment.
| command() | |
| .args(["dylint", "--all", "--no-build"]) | |
| .assert() | |
| .success() | |
| .stderr( | |
| predicate::str::contains("\nwarning: `unwrap`s that could be combined\n") | |
| .and(predicate::str::contains("Compiling").not()), | |
| ); |
Please remove this. Again, it just anticipates a bug fix.
| } | ||
|
|
||
| #[test] | ||
| fn libraries_dir_unset_preserves_workspace_cache() { |
There was a problem hiding this comment.
I don't see a lot of value in this test. I recommend removing it, unless you feel strongly that it should stay.
|
Addressed all nine comments in 58b7410, as a separate commit without rewriting the original. Applied the suggested wording to CLI help and all three READMEs. Simplified the temporary directory and command setup, reordered and separated the target/copy/release checks, and removed the Validation: five integration tests, the two affected README synchronization tests, formatting, Clippy, and workspace custom lints passed. An unrelated utility-README formatting check needs AI acting on behalf of @dblock. |
|
Thanks, @dblock. Could you also please collapse the changes into one commit? (Sorry, I should have asked for that already.) |
Add the opt-in absolute DYLINT_LIBRARIES_DIR override while preserving the workspace-local default, per-toolchain release layout, Cargo build-directory configuration, and library-copy behavior. Cover cross-workspace warm reuse, both copy modes, explicit Cargo build directories, and invalid overrides, and document the setting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
58b7410 to
46e8792
Compare
Certainly, I was just trying to make it easier to review the diff on top of the diff. Done. |
Summary
Add the opt-in
DYLINT_LIBRARIES_DIRoverride proposed in #2142. It must be an absolute path and is created as needed. Leaving it unset preserves the existing workspace-local default.Keep the existing layout:
<directory>/<toolchain>/release, where<directory>is eitherDYLINT_LIBRARIES_DIRor the workspace'starget/dylint/libraries. There are no package, version, or source-identity subdirectories. Explicit Cargo build-directory configuration and both library-copy modes retain their existing behavior.Compiler-driver caches are unchanged. This PR does not introduce
DYLINT_CACHE_DIR, change default cache locations, or migrate existing caches.Why This Matters
Every fresh consumer worktree currently recompiles the same lint libraries. This becomes noticeable when creating worktrees frequently for separate tasks.
Earlier local measurements on Apple Silicon using dylint-spellcheck v0.1.0, with source/dependency downloads and the Rust toolchain already cached, showed two fresh consumer directories taking 12.419 s and 12.149 s, compiling 66 crates each time.
Another fresh consumer directory using the warm shared library cache took 0.754 s and compiled zero crates: about 11.5 seconds saved per new worktree, or roughly 16x faster lint setup. These measurements cover lint-library setup, not compilation of a real consumer project.
There is duplicated disk usage too: four existing already-tested worktrees each contained 271 MB under
target/dylint/libraries. Opting into a directory outside the workspace allows those builds to be reused across worktrees, retained through consumercargo clean, and cached separately in CI.Source Collisions
Source partitioning is outside this PR's scope. Same-name libraries using one toolchain can replace each other, potentially triggering rebuilds or leaving
--no-buildusing the wrong revision. The existing error for multiple libraries with the same output path in one invocation remains unchanged.Validation
Formatting, workspace custom lints, Clippy, five targeted integration tests, the two affected README synchronization tests, and Markdown formatting passed locally on macOS. Tests cover cross-workspace warm reuse, invalid overrides, explicit Cargo build directories, and both library-copy modes.
An additional utility-README formatting check could not run locally because
cargo-rdmeis not installed; no utility READMEs are changed.Closes #2142.
AI-authored on behalf of @dblock.