Detect property to exclude Cargo dependencies #1421

zahidblackduck · 2025-05-06T13:42:42Z

JIRA Ticket

IDETECT-4326

Description

This merge request introduces a feature to exclude specific types of dependencies in the Cargo detector, providing more control over which dependencies are included in detect scans. The supported types are:

Normal: Regular dependencies needed for runtime.
Dev: Development dependencies, such as test libraries.
Build: Build-related dependencies.
Proc-Macro: Procedural macro dependencies.

A new property has been added to configure the exclusion of these dependencies:

--detect.cargo.dependency.types.excluded=NONE,NORMAL,DEV,BUILD,PROC_MACRO

By default, all dependencies are included (NONE). Specific types can be excluded by specifying the appropriate values.

Implementation Details

To support this feature, an enum (CargoDependencyType) has been introduced to represent the dependency types. This logic has been implemented across both Cargo CLI Detector and Cargo Lockfile Detector:

Cargo CLI Detector: Uses the cargo tree command with native options to exclude dependencies. For example, excluding development dependencies is done by adding the --edges no-dev flag to the command. This solution leverages the CLI's built-in functionality with minimal post-processing.
Cargo Lockfile Detector: Since Cargo.lock does not include dependency types, the lockfile detector reads and parses the Cargo.toml file. ~~It collects the [dev-dependencies] and [build-dependencies] sections, filtering out matching dependencies based on the exclusion configuration.~~ It collects the sections that are to be included from the Cargo.toml file, then collects the corresponding transitive dependencies from the Cargo.lock using DFS traversal.

.../src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoCliExtractor.java

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java

devmehtabd · 2025-05-07T19:17:51Z

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java

+        List<CargoLockPackageData> filteredPackages = cargoLockPackageDataList;
+        String cargoTomlContents = FileUtils.readFileToString(cargoTomlFile, StandardCharsets.UTF_8);
+
+        if (cargoDetectableOptions != null) {


Instead of checking on cargoDetectableOptions object, can we check on dependencyTypeFilter instead? There could be more properties added to this detector in future not related to dependency exclusion.

Ah, you're right. The actual filtering is handled by dependencyTypeFilter. This null check is just to verify whether any options were passed at all. The filtering logic using dependencyTypeFilter is applied later in the CargoTomlParser#parseDependenciesToExclude(..) method.

I understand that, my only concern is that if in future lets say we add some new property for excluding workspaces as an example then those options will also be passed in CargoDetectableOptions, so we will make this unnecessary call to cargoTomlParser which is not required if no dependency exclusion is involved. I hope I made sense.

The current code setup doesn't call the methods of CargoTomlParser if no dependency exclusion is involved. Nonetheless, I aim to address the issue that you pointed out in any future enhancement of cargo related issue.

I know, currently your code is not getting executed if cargoDetectableOptions is null. It is not guaranteed that you are going to work on this in any future enhancement related to Cargo and it could be easily missed. I believe we should address this now, its a simple change and not a complicated ask.

Okay, sure. I'll update accordingly.

devmehtabd · 2025-05-07T19:20:14Z

src/main/java/com/blackduck/integration/detect/configuration/DetectProperties.java

+            .setInfo("Cargo Dependency Types Excluded", DetectPropertyFromVersion.VERSION_10_5_0)
+            .setHelp(
+                "A comma-separated list of dependency types that will be excluded.",
+                "If DEV is excluded, the Cargo CLI Detector will exclude 'dev' dependencies when parsing the output of cargo tree."


Should be changed, since this is applicable to Lockfile Detector as well.

It's a good find. I'll update.

devmehtabd · 2025-05-07T19:22:26Z

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java

@@ -1,16 +1,22 @@
 package com.blackduck.integration.detectable.detectables.cargo.parse;

-import java.util.Optional;
+import java.util.*;


Another wildcard import statement.

cpottsbd · 2025-05-13T19:06:16Z

src/main/java/com/blackduck/integration/detect/configuration/DetectProperties.java

+            .setInfo("Cargo Dependency Types Excluded", DetectPropertyFromVersion.VERSION_10_5_0)
+            .setHelp(
+                "A comma-separated list of dependency types that will be excluded.",
+                "The Cargo CLI Detector uses cargo tree flags to exclude the specified types, while the Cargo Lockfile Detector filters dependencies by reading Cargo.toml. For example, passing DEV will skip [dev-dependencies] from detection."


I might be inclined to pass the full parameter entry for the example?
i.e. "For example, passing DETECT_CARGO_DEPENDENCY_TYPES_EXCLUDED = DEV will skip [dev-dependencies] from detection."

Thanks. Updated as per the review.

…toml

…IDETECT-4326

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java

…allback

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java

andrian-sevastyanov · 2025-05-20T20:39:59Z

I have an idea that could help resolve the special cases we have been discussing. For Cargo.lock, when excluding dependencies we could try something like this:

Go through sections of Cargo.toml to determine which sections should be included
For all included sections, figure out what the root dependencies are
Use the lock file to figure out the transitive dependencies for root dependencies from step 2
The final result is then combination of step 2 and 3.

I may have missed something, but I think it's worth considering at the very least.

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java

...lackduck/integration/detectable/detectables/cargo/transform/CargoLockPackageTransformer.java

andrian-sevastyanov · 2025-05-20T21:58:00Z

.../src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoCliExtractor.java

+        exclusionMap.put(CargoDependencyType.NORMAL, "no-normal");
+        exclusionMap.put(CargoDependencyType.BUILD, "no-build");
+        exclusionMap.put(CargoDependencyType.DEV, "no-dev");
+        exclusionMap.put(CargoDependencyType.PROC_MACRO, "no-proc-macro");


Could you describe situations in which PROC_MACRO exclusion is useful? And I guess it's not applicable to lock file extractions?

…IDETECT-4326

zahidblackduck · 2025-05-22T12:56:39Z

I have an idea that could help resolve the special cases we have been discussing. For Cargo.lock, when excluding dependencies we could try something like this:

Go through sections of Cargo.toml to determine which sections should be included

For all included sections, figure out what the root dependencies are

Use the lock file to figure out the transitive dependencies for root dependencies from step 2

The final result is then combination of step 2 and 3.

I may have missed something, but I think it's worth considering at the very least.

Interesting idea. Further discussion might help on the implementation strategy.

…IDETECT-4326

…cy exclusion

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java

andrian-sevastyanov · 2025-06-24T15:13:33Z

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java

+        Deque<NameVersion> queue = new ArrayDeque<>(dependenciesToExclude);
+        while (!queue.isEmpty()) {
+            NameVersion current = queue.pop();
+            CargoLockPackageData currentPkg = findPackageByNameVersion(current, packages);


This find is done inside of a loop so it might be worth it to do some preprocessing for faster lookups.

This may still be a concern for big projects. You could try indexing packages by name for faster lookups before starting loops that call this method. There are a couple such loops.

So, are you suggesting to construct a HashMap of package names beforehand? I mean the findPackageByNameVersion method matches for versions. For example, which 7.0.0 should match version which 7.0.2. And there can be another which 8.0.1 to which the first one should not match.

Yeah, that's the idea. It can be a map of names to multiple name/versions.
That would allow us to quickly find all possible versions for a given dependency instead of iterating through the entire list of dependencies for every single lookup.

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java

…IDETECT-4326

zahidblackduck · 2025-06-26T12:12:26Z

I have an idea that could help resolve the special cases we have been discussing. For Cargo.lock, when excluding dependencies we could try something like this:

Go through sections of Cargo.toml to determine which sections should be included

For all included sections, figure out what the root dependencies are

Use the lock file to figure out the transitive dependencies for root dependencies from step 2

The final result is then combination of step 2 and 3.

I may have missed something, but I think it's worth considering at the very least.

The latest changes of the Cargo Lockfile detector dependency exclusion strategy has adopted the above approach.

zahidblackduck added 9 commits April 23, 2025 11:55

property to exclude dev dependencies in cargo cli detector add

10883f5

cargo cli extractor dev dependencies exclusion add

0354a8e

cargo lockfile detector dependency exclusion add

2830904

detect rule factory edit

1ac92d4

cargo version match util update

5344176

cargo version match util add

fba1188

cargo lockfile dev,build dependency exclusion add

f6e48d9

Merge conflict resolve with master

b66b9f7

cargo lock file dependency transformer refactor

37c4ed6

zahidblackduck marked this pull request as draft May 6, 2025 13:43

zahidblackduck self-assigned this May 6, 2025

zahidblackduck requested review from dterrybd, devmehtabd, shantyk and andrian-sevastyanov May 6, 2025 13:44

andrian-sevastyanov reviewed May 6, 2025

View reviewed changes

.../src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoCliExtractor.java Outdated Show resolved Hide resolved

andrian-sevastyanov requested a review from cpottsbd May 6, 2025 14:26

andrian-sevastyanov reviewed May 6, 2025

View reviewed changes

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java Outdated Show resolved Hide resolved

andrian-sevastyanov reviewed May 6, 2025

View reviewed changes

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java Outdated Show resolved Hide resolved

andrian-sevastyanov reviewed May 6, 2025

View reviewed changes

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java Outdated Show resolved Hide resolved

zahidblackduck added 2 commits May 7, 2025 15:29

wildcard imports cleared

9e1e4a4

method name refactor to express actual intent

95d1bac

devmehtabd reviewed May 7, 2025

View reviewed changes

doc description updated for new detect properties

1293fb9

cpottsbd reviewed May 13, 2025

View reviewed changes

zahidblackduck added 3 commits May 15, 2025 18:39

handle dependency exclusion if present in multiple sections of Cargo.…

1a0e93a

…toml

handle dependency exclusion if present in multiple sections of Cargo.…

0d8f6f2

…toml

properties doc updated with an example usage

d2a8a6c

zahidblackduck requested review from andrian-sevastyanov and devmehtabd May 15, 2025 12:57

Merge remote-tracking branch 'origin/master' into dev/zahidblackduck/…

5eb9b16

…IDETECT-4326

cpottsbd approved these changes May 15, 2025

View reviewed changes

andrian-sevastyanov reviewed May 15, 2025

View reviewed changes

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java Outdated Show resolved Hide resolved

refactor dependency exclusion to use NameVersion without precedence f…

d2be399

…allback

zahidblackduck requested a review from andrian-sevastyanov May 16, 2025 12:46

zahidblackduck added 2 commits May 16, 2025 18:49

remove code comment and empty line

83ebdf1

refactor condition to check actual dependency exclusion filter

ae1ee8e

devmehtabd approved these changes May 20, 2025

View reviewed changes

andrian-sevastyanov reviewed May 20, 2025

View reviewed changes

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java Outdated Show resolved Hide resolved

andrian-sevastyanov reviewed May 20, 2025

View reviewed changes

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java Show resolved Hide resolved

andrian-sevastyanov reviewed May 20, 2025

View reviewed changes

...lackduck/integration/detectable/detectables/cargo/transform/CargoLockPackageTransformer.java Outdated Show resolved Hide resolved

andrian-sevastyanov reviewed May 20, 2025

View reviewed changes

zahidblackduck added 2 commits May 22, 2025 14:14

Merge remote-tracking branch 'origin/master' into dev/zahidblackduck/…

383b99e

…IDETECT-4326

parse dependencies to exclude map key change to NameVersion

204b05f

zahidblackduck added 4 commits May 27, 2025 14:39

Merge remote-tracking branch 'origin/master' into dev/zahidblackduck/…

538bc03

…IDETECT-4326

Merge remote-tracking branch 'origin/master' into dev/zahidblackduck/…

537760c

…IDETECT-4326

transitive dependencies resolve for cargo lock file detector dependen…

f1be648

…cy exclusion

version util update to follow cargo version matching properly

f3130f8

zahidblackduck requested a review from andrian-sevastyanov June 23, 2025 10:45

andrian-sevastyanov requested changes Jun 24, 2025

View reviewed changes

zahidblackduck added 3 commits June 25, 2025 13:00

Merge remote-tracking branch 'origin/master' into dev/zahidblackduck/…

9955084

…IDETECT-4326

exclusion strategy update for cargo lockfile detector

44a70b2

exclusion logic re-implement in preemptive approach

26ec154

zahidblackduck requested a review from andrian-sevastyanov June 26, 2025 12:03

andrian-sevastyanov approved these changes Jun 26, 2025

View reviewed changes

Detect property to exclude Cargo dependencies #1421

Are you sure you want to change the base?

Detect property to exclude Cargo dependencies #1421

Uh oh!

Conversation

zahidblackduck commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zahidblackduck May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

andrian-sevastyanov commented May 20, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zahidblackduck commented May 22, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zahidblackduck commented Jun 26, 2025

Uh oh!

Uh oh!

zahidblackduck commented May 6, 2025 •

edited

Loading

zahidblackduck May 8, 2025 •

edited

Loading