Skip to content

Support a shared directory for compiled lint libraries - #2179

Merged
smoelius merged 1 commit into
trailofbits:masterfrom
dblock:shared-libraries-dir
Nov 19, 2026
Merged

smoelius merged 1 commit into
trailofbits:masterfrom
dblock:shared-libraries-dir

Conversation

@dblock

@dblock dblock commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add the opt-in DYLINT_LIBRARIES_DIR override proposed in #2142. It must be an absolute path and is created as needed. Leaving it unset preserves the existing workspace-local default.

Keep the existing layout: <directory>/<toolchain>/release, where <directory> is either DYLINT_LIBRARIES_DIR or the workspace's target/dylint/libraries. There are no package, version, or source-identity subdirectories. Explicit Cargo build-directory configuration and both library-copy modes retain their existing behavior.

Compiler-driver caches are unchanged. This PR does not introduce DYLINT_CACHE_DIR, change default cache locations, or migrate existing caches.

Why This Matters

Every fresh consumer worktree currently recompiles the same lint libraries. This becomes noticeable when creating worktrees frequently for separate tasks.

Earlier local measurements on Apple Silicon using dylint-spellcheck v0.1.0, with source/dependency downloads and the Rust toolchain already cached, showed two fresh consumer directories taking 12.419 s and 12.149 s, compiling 66 crates each time.

Another fresh consumer directory using the warm shared library cache took 0.754 s and compiled zero crates: about 11.5 seconds saved per new worktree, or roughly 16x faster lint setup. These measurements cover lint-library setup, not compilation of a real consumer project.

There is duplicated disk usage too: four existing already-tested worktrees each contained 271 MB under target/dylint/libraries. Opting into a directory outside the workspace allows those builds to be reused across worktrees, retained through consumer cargo clean, and cached separately in CI.

Source Collisions

Source partitioning is outside this PR's scope. Same-name libraries using one toolchain can replace each other, potentially triggering rebuilds or leaving --no-build using the wrong revision. The existing error for multiple libraries with the same output path in one invocation remains unchanged.

Validation

Formatting, workspace custom lints, Clippy, five targeted integration tests, the two affected README synchronization tests, and Markdown formatting passed locally on macOS. Tests cover cross-workspace warm reuse, invalid overrides, explicit Cargo build directories, and both library-copy modes.

An additional utility-README formatting check could not run locally because cargo-rdme is not installed; no utility READMEs are changed.

Closes #2142.

AI-authored on behalf of @dblock.

@dblock
dblock requested a review from smoelius as a code owner November 13, 2026 19:07
@CLAassistant

CLAassistant commented Nov 13, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@dblock
dblock force-pushed the shared-libraries-dir branch 2 times, most recently from e8031c4 to 3bc93fa Compare November 13, 2026 20:48
@dblock dblock mentioned this pull request Nov 13, 2026
@dblock
dblock marked this pull request as draft November 13, 2026 22:37
@smoelius

Copy link
Copy Markdown
Collaborator

@dblock Thanks very much for working on this.

Partition shared builds as <directory>/<toolchain>/<package>-<identity-hash>/release/.

Why the <package>-<identity-hash> component? What problem do you foresee?

@dblock

dblock commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Why the <package>-<identity-hash> component? What problem do you foresee?

A fork or a separate local checkout that hasn't been released. Importantly, --no-build could silently load the wrong revision. The actual layout is /-/. For Git dependencies, the hash identifies the repository, resolved commit, and package path—not the consuming workspace—so different consumer worktrees still share the cache.

@dblock
dblock force-pushed the shared-libraries-dir branch 2 times, most recently from 1660c8e to dd0ca1d Compare November 17, 2026 03:43
@smoelius

Copy link
Copy Markdown
Collaborator

A fork or a separate local checkout that hasn't been released. Importantly, --no-build could silently load the wrong revision. The actual layout is /-/. For Git dependencies, the hash identifies the repository, resolved commit, and package path—not the consuming workspace—so different consumer worktrees still share the cache.

If I understand correctly, this is a problem with how target/dylint/libraries is organized today.

As such, I would rather disentangle the fix from this PR and #2180.

That would also simplify the logic because shared DYLINT_LIBRARIES_DIR directories and directories under target could be treated the same (i.e., I think the shared field should become unnecessary).

Does that make sense?

Also, is the collision problem something you are running into? Or were you just trying to be proactive?

@dblock

dblock commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Yes, that makes sense. The problem I’m actually encountering is repeated compilation across consumer worktrees; collision prevention was proactive.

Would you prefer that I drop source partitioning from both PRs for now, preserve the existing cache layout and behavior, and leave collision handling for separate work? That should also let us remove the shared distinction.

So the plan would be to

  1. Close this Support a shared directory for compiled lint libraries #2179 PR because it introduces DYLINT_LIBRARIES_DIR as an opt-in setting, but Add an opt-in DYLINT_HOME for storing drivers #2180 immediately replaces it with DYLINT_CACHE_DIR.
  2. Simplify Add an opt-in DYLINT_HOME for storing drivers #2180 to share caches through DYLINT_CACHE_DIR, defaulting to ~/.dylint/cache, drop library source partitioning, defer source-collision handling.
  3. Defer separating cached libraries by source identity to a follow-up PR.

@smoelius

Copy link
Copy Markdown
Collaborator

Would you prefer that I drop source partitioning from both PRs for now, preserve the existing cache layout and behavior, and leave collision handling for separate work? That should also let us remove the shared distinction.

Yes.

  1. Close this Support a shared directory for compiled lint libraries #2179 PR because it introduces DYLINT_LIBRARIES_DIR as an opt-in setting, but Use a unified home cache with automatic legacy migration #2180 immediately replaces it with DYLINT_CACHE_DIR.
  2. Simplify Use a unified home cache with automatic legacy migration #2180 to share caches through DYLINT_CACHE_DIR, defaulting to ~/.dylint/cache, drop library source partitioning, defer source-collision handling.
  3. Defer separating cached libraries by source identity to a follow-up PR.

If it's not a huge ask, could we keep this PR and update #2180 once this one is settled?

I actually find them easier to review as separate PRs.

@dblock
dblock force-pushed the shared-libraries-dir branch from dd0ca1d to d229697 Compare November 18, 2026 01:51
@dblock dblock changed the title Support a shared directory for compiled lint libraries Share compiler drivers and lint libraries in a unified cache Nov 18, 2026
@dblock
dblock marked this pull request as ready for review November 18, 2026 01:53
@dblock

dblock commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

If it's not a huge ask, could we keep this PR and update #2180 once this one is settled?

I actually find them easier to review as separate PRs.

Whatever you need to get the changes merged, don't hold back asking! This is done, I updated this PR with what we discussed (I hope). Thanks for taking the time for the review.

@smoelius

Copy link
Copy Markdown
Collaborator

Sorry. Rereading our discussion, I think I may have been unclear.

I was hoping that this PR could be what it was before, just without collision handling.

So, essentially, the addition of the DYLINT_LIBRARIES_DIR feature, but without the new <package>/<version>-<source-hash>/<toolchain> layout.

Would that be possible?

@dblock
dblock force-pushed the shared-libraries-dir branch from d229697 to 651e25f Compare November 18, 2026 12:28
@dblock dblock changed the title Share compiler drivers and lint libraries in a unified cache Support a shared directory for compiled lint libraries Nov 18, 2026
@dblock

dblock commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Sorry. Rereading our discussion, I think I may have been unclear.

I was hoping that this PR could be what it was before, just without collision handling.

So, essentially, the addition of the DYLINT_LIBRARIES_DIR feature, but without the new <package>/<version>-<source-hash>/<toolchain> layout.

Would that be possible?

Updated. The consolidation PR will remove the setting ;)

@smoelius smoelius left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please let me know if I have misunderstood your changes in any way.

Comment thread cargo-dylint/src/main.rs Outdated
Comment on lines +28 to +30
DYLINT_LIBRARIES_DIR (default: <workspace target>/dylint/libraries) is an absolute path where
Dylint builds lint libraries, using <directory>/<toolchain>/release. Set it to share builds
across projects and worktrees. It does not isolate libraries by source.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
DYLINT_LIBRARIES_DIR (default: <workspace target>/dylint/libraries) is an absolute path where
Dylint builds lint libraries, using <directory>/<toolchain>/release. Set it to share builds
across projects and worktrees. It does not isolate libraries by source.
DYLINT_LIBRARIES_DIR (default: <target directory>/dylint/libraries) is an absolute path where
Dylint builds lint libraries, using <directory>/<toolchain>/release. Set it to share builds
across projects.

These are nits. I think "target directory" is more clear than "workspace target"; "worktrees" is a Git concept, and IMO tangential; and the last sentence anticipates a bug fix and will just get removed anyway.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI on behalf of @dblock: Applied your suggested help text in 58b7410, using “target directory” and removing the worktree discussion and source-isolation caveat.

Comment thread cargo-dylint/README.md Outdated

Dylint downloads and builds each entry, similar to how Cargo downloads and builds a dependency. The resulting `target/release` directories are searched for files with names of the form that Dylint recognizes (see [Library requirements] under [How Dylint works]).

By default, these builds use `<workspace target>/dylint/libraries/<toolchain>/release`. Set `DYLINT_LIBRARIES_DIR` to an absolute path to reuse builds across projects or worktrees under `<directory>/<toolchain>/release` instead; the directory is created as needed. Same-name libraries using one toolchain can replace each other, and `--no-build` does not validate their source.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
By default, these builds use `<workspace target>/dylint/libraries/<toolchain>/release`. Set `DYLINT_LIBRARIES_DIR` to an absolute path to reuse builds across projects or worktrees under `<directory>/<toolchain>/release` instead; the directory is created as needed. Same-name libraries using one toolchain can replace each other, and `--no-build` does not validate their source.
By default, these builds use `<target directory>/dylint/libraries/<toolchain>/release`. Set `DYLINT_LIBRARIES_DIR` to an absolute path to reuse builds under `<directory>/<toolchain>/release` instead. The directory is created as needed.

For reasons similar to the help message. Also, "projects...under <directory>/<toolchain>/release" sounds like that is where the projects live.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI on behalf of @dblock: Applied your suggested paragraph in 58b7410 and synchronized all three README copies.

Comment on lines +169 to +170
let cache = tempdir().unwrap();
let libraries_directory = cache.path().join("shared libraries");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let cache = tempdir().unwrap();
let libraries_directory = cache.path().join("shared libraries");
let libraries_directory = tempdir().unwrap();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI on behalf of @dblock: Changed libraries_directory to a direct TempDir in 58b7410 and updated its uses to call .path().

.unwrap();
let target_directory = fixture.path().join("target");

let command = || {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let command = || {

I'm recommending the --no-build part of this test go away; hence, command should be usable directly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI on behalf of @dblock: Removed the command-producing closure in 58b7410; the test now constructs and configures the command directly.

Comment on lines +197 to +198
command
};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
command
};

See above.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI on behalf of @dblock: Updated the call site in 58b7410 to use command.args(...) directly rather than invoking a closure.

Comment on lines +212 to +217
let release_directory = libraries_directory.join(&toolchain).join("release");
assert!(
release_directory
.join(library_plain_filename("collapsible_unwrap"))
.is_file()
);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move the release_directory check to after the copied_library_directory check. IMO, that overall order is more clear, i.e., the target directory is checked first, then where the library is copied to, then where the library ultimately lands.

Please leave a newline after this let and assert! are moved. Please also separate them from the copied_library_directory check with a newline.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI on behalf of @dblock: Reordered the checks in 58b7410 to target directory, copied-library directory, then final release library, with blank lines separating each block.

Comment on lines +232 to +239
command()
.args(["dylint", "--all", "--no-build"])
.assert()
.success()
.stderr(
predicate::str::contains("\nwarning: `unwrap`s that could be combined\n")
.and(predicate::str::contains("Compiling").not()),
);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
command()
.args(["dylint", "--all", "--no-build"])
.assert()
.success()
.stderr(
predicate::str::contains("\nwarning: `unwrap`s that could be combined\n")
.and(predicate::str::contains("Compiling").not()),
);

Please remove this. Again, it just anticipates a bug fix.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI on behalf of @dblock: Removed the --no-build portion of the shared-cache test in 58b7410.

}

#[test]
fn libraries_dir_unset_preserves_workspace_cache() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a lot of value in this test. I recommend removing it, unless you feel strongly that it should stay.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI on behalf of @dblock: Removed libraries_dir_unset_preserves_workspace_cache in 58b7410 as suggested.

@dblock

dblock commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Addressed all nine comments in 58b7410, as a separate commit without rewriting the original.

Applied the suggested wording to CLI help and all three READMEs. Simplified the temporary directory and command setup, reordered and separated the target/copy/release checks, and removed the --no-build checks and the unset-directory test.

Validation: five integration tests, the two affected README synchronization tests, formatting, Clippy, and workspace custom lints passed. An unrelated utility-README formatting check needs cargo-rdme, which is not installed locally.

AI acting on behalf of @dblock.

@smoelius

Copy link
Copy Markdown
Collaborator

Thanks, @dblock. Could you also please collapse the changes into one commit? (Sorry, I should have asked for that already.)

Add the opt-in absolute DYLINT_LIBRARIES_DIR override while preserving the workspace-local default, per-toolchain release layout, Cargo build-directory configuration, and library-copy behavior. Cover cross-workspace warm reuse, both copy modes, explicit Cargo build directories, and invalid overrides, and document the setting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dblock
dblock force-pushed the shared-libraries-dir branch from 58b7410 to 46e8792 Compare November 19, 2026 13:27
@dblock

dblock commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, @dblock. Could you also please collapse the changes into one commit? (Sorry, I should have asked for that already.)

Certainly, I was just trying to make it easier to review the diff on top of the diff. Done.

@smoelius
smoelius added this pull request to the merge queue Nov 19, 2026
Merged via the queue into trailofbits:master with commit 96243ea Nov 19, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Installing/caching lints

3 participants