feat: add AI evaluation for SA-01.01, SA-02.01, SA-03.01, SA-03.02 - #480
Open
vinayada1 wants to merge 1 commit into
Open
feat: add AI evaluation for SA-01.01, SA-02.01, SA-03.01, SA-03.02#480vinayada1 wants to merge 1 commit into
vinayada1 wants to merge 1 commit into
Conversation
Add optional AI review for design, interface, security assessment, and threat-model documentation explicitly declared in Security Insights. Fetch bounded same-repository text artifacts at their declared refs without repository-wide discovery or downstream link traversal. Defer incomplete evidence and AI failures to manual review. Require human confirmation of interface pass recommendations while retaining the original AI evidence, and calibrate risk assessment grading around explicit likelihood-and-impact prioritization. Apply the published-release gate to design documentation and add retrieval, prompt, result-handling, and opt-in live replay tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Vinaya Damle <vinayada1@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add optional AI evaluation paths for four OSPS security-assessment requirements:
Scope
This PR's new AI paths only evaluate evidence explicitly declared in Security Insights; they do not look for additional evidence on their own. Design review uses the declared detailed guide; interface review uses the detailed and quickstart guides; security-assessment and threat-model review use the declared self and third-party assessment evidence.
The AI evaluates the contents of those artifacts rather than treating a filename or declaration as proof of sufficient documentation. This PR does not add repository-wide evidence discovery, source-code scanning, arbitrary external fetching, or downstream-link traversal. Existing deterministic checks remain separate from these new AI paths.
Evidence and result handling
Important SA-02.01 limitation
Every SA-02.01 AI pass recommendation is returned as NeedsReview/Low, requiring human confirmation. Live grading sometimes accepted insufficient interface documentation despite prompt tightening, so this is enforced in code, not merely requested by the prompt.
The original AI verdict, explanation, citations, and supplied material remain in the evidence for human review. AI Failed and NeedsReview responses retain their normal result handling. This intentionally also defers otherwise valid AI pass recommendations.
Non-AI behavior changes
SA-01.01 now applies the published-release applicability gate already used by the other three checks and guards optional SI documentation fields. Its deterministic missing-design result now explicitly uses Medium confidence. Release detection still uses GitHub Releases, not tags alone or releases distributed elsewhere.
Validation
go test -race ./data ./evaluation_plans/osps/sec_assessment ./evaluation_plans/reusable_steps -count=1.golangci-lint,make -B build, andgit diff --checkafter the final policy change..github/scripts/ci.shintegration with all four supported catalog IDs.go test ./... -count=1andgolangci-lint run ./...passed on the updated upstream before the final SA-02 policy change; the related checks above were rerun afterward.gpt-5.4-minithree times each: all nine final scanner results matched the expected Meshery SA-03.01 NeedsReview, FDC3-Sail SA-01.01 Passed, and FDC3-Sail SA-02.01 NeedsReview outcomes.The committed opt-in live test exercises production grading with caller-supplied captured fixtures; it does not replace full scanner retrieval tests or run in the normal unit suite. These finite examples do not establish general model accuracy. Existing SI reader rate limiting and legacy-schema incompatibilities remain outside this PR.