fix(conformance): separate not-applicable checks from met and from review - #662
Merged
Conversation
…view A check that measures something the document does not contain was reported as met: no packages to measure, no files, no parts to relate. That ranked a document declaring less above one declaring more. An AI SBOM whose card names no dataset collected three format checks for having nothing to measure, while one that names three datasets was measured on them and could lose points — the same inversion #656 fixed for the dataset producer field. It also turned into a mandatory failure. #661 removed the generator's root dependency edge, which was keyed on an id nothing in the document defined, so a model with no declared datasets had no edges left and failed a required check for having no parts to relate. Mark those checks naKind "not-applicable" and keep them out of both sides of every fraction: the verdict tally, the advisory coverage base, the format tally, and the AI profile rollup. Neither met nor a gap a reader can close, and not a review item either, since a reviewer has nothing to look at. The reports and the UI carry their own label and a muted state; a document with parts and no edges still fails.
test-postprocess.sh pinned the earlier reading of an empty package denominator as met. It stays not-uncovered, which is what that check was defending, but it is no longer credited either. The data-only case reads the two coverage checks alone: removing the model from an AI SBOM also takes the 1.7 spec version out of its allowed range, which fails the document for an unrelated reason.
This was referenced Aug 10, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A check that measures something the document does not contain was counted as
met — "no packages to measure", "no file components", "nothing to relate". That
inverts the incentive: a document declaring fewer parts scores higher than one
declaring them and being measured on them. Measured on four public AI models,
a card naming no dataset collected three format checks for having nothing to
measure, while the one naming three datasets was measured and could lose points.
It is the same inversion #656 fixed for the dataset producer field.
The empty case had also become a mandatory failure. #661 removed the generator's
root dependency edge (keyed on an id nothing in the document defines), so a model
with no declared datasets had no edges left and failed a required check for
having no parts to relate.
Change
Twelve checks now carry
naKind: "not-applicable"when their subject is empty,and such a check leaves both sides of every fraction:
It is not folded into the existing review category: a review item has something
to judge and no automated source for it, while this one gives a reviewer nothing
to look at. Reports and the UI label it separately (
N/A, muted tone, sortedbelow review rows), and the summary line carries the count. A document with parts
and no edges still fails.
Verification
Four public AI models, same settings, before and after:
The 17/17 was the bug: eleven of those checks had nothing to measure. The 2026
minimum elements (17/23 vs 16/23) and G7 (32/51 vs 23/51) are unchanged, so the
baselines that do measure something still say what they said.
tests/test-aibom.sh: 262 passed, 0 failed (5 new assertions)tests/test-web-ui.sh: 118 passed, 0 failedtscand lint cleanshellcheck --severity=warningclean