refactor(agents)!: consolidate supply-chain reviewers and align data-science-engineering vocabulary - #2720
Open
Bill Berry (WilliamBerryiii) wants to merge 21 commits into
Open
Conversation
…viewer Port the Codebase Profiler, Supply Chain Skill Assessor, and Finding Deep Verifier stages from the duplicate Supply Chain Reviewer so its removal costs no capability. Reconcile the frontmatter agents list with what the body dispatches, and define the status and verdict translation into the nine-section SSSC report contract. Report Generator is excluded; SSSC Reviewer keeps authoring its own report.
SSSC Reviewer now owns supply-chain posture review after P01 ported its assessment and verification pipeline. Remove the duplicate agent, drop it from the security and hve-core-all packages, repoint the code-review supply-chain fork to sssc-reviewer, and clear its eval registry entries. Reconcile the project-planning package, which shipped SSSC Reviewer without the subagents it now dispatches. Also requalify two stale references to the removed agent.
Pure git mv with no content changes so renames stay detectable and reviewable. Package becomes data-science-engineering, the coach agent file and its slug-derived eval files take the new stem, five skills drop the ds- prefix, and the foundation skill is renamed. DS_CATALOG_V1, the feasibility interchange identifier, and the .copilot-tracking/ds/ session root are deliberately unchanged. References are rewritten in the next commit.
…g rename Update marketplace identity and component paths, instruction applyTo globs, skill names and prose, Python package names, cross-package references, eval registries, hand-written docs, and repository configuration to the new package, skill, and agent names. Rename the coaching disclaimer heading to Data Science and Engineering Coaching. Rewrite the moved security docs page as an accurate SSSC Reviewer page with the three-stage pipeline. DS_CATALOG_V1, its schema id and filenames, the feasibility interchange identifier, the session schema_version, and the .copilot-tracking/ds/ root are unchanged.
…ncerns The package shipped privacy-standards, rai-standards, rai-planner, and telemetry-foundations while the coach listed only the seven data-science skills and its stop rules forbade crossing that boundary, contradicting the registry it loads. rai-standards was routed nowhere. Define the three concerns by trigger in the job registry, make the registry their single source of truth, replace the count-based stop rule with an authority-based one, and refresh the coach eval expectations. Concerns surface and offer choices; the durable-write scan gate remains the only blocking gate.
…g-consolidation # Conflicts: # docs/plugins/project-planning.md # docs/reference/README.md # docs/reference/agents/README.md # docs/reference/agents/security/supply-chain-reviewer.md # docs/reference/prompts/README.md # docs/reference/prompts/data-science-engineering/synth-data-generate.md # docs/reference/skills/README.md
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF ScorecardScorecard details
Scanned Files
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2720 +/- ##
==========================================
+ Coverage 83.17% 84.60% +1.43%
==========================================
Files 180 95 -85
Lines 32201 10763 -21438
Branches 25 25
==========================================
- Hits 26782 9106 -17676
+ Misses 5416 1654 -3762
Partials 3 3
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Contributor
Eval Execution✅ Status: Passed — no merge-blocking failures (105 advisory assertion failure(s) present)
|
…package Two test fixtures referenced the old package name and were missed by the reference sweep. The architecture-diagrams catalog fixture path splits the package segment across its own line, so the path rewrite did not reach it. The marketplace closure test matched the package by its old name.
…ailable Porting the assessment pipeline made subagent dispatch mandatory, including a hard stop when no skill remained. A reviewer installed without its subagents produced no findings at all, which is worse than the prior inline behavior and matches the packaging risk raised as PC-010. Each ported stage now degrades to direct assessment and records the reduced rigor in Limitations, so a report with findings and severities is always produced.
Jamie Kim (jkim323)
requested changes
Aug 17, 2026
Jamie Kim (jkim323)
left a comment
Jamie Kim (jkim323)
left a comment
Collaborator
There was a problem hiding this comment.
Left a couple things for you to look at. Happy to review back - thanks for working on this consolidation effort!
…g-consolidation # Conflicts: # .github/plugin/marketplace.json # docs/plugins/experimental.md # docs/plugins/hve-core-all.md # docs/plugins/security.md # docs/reference/README.md # docs/reference/agents/README.md # docs/reference/agents/data-science-engineering/data-science-engineering-coach.md # docs/reference/agents/security/supply-chain-reviewer.md # docs/reference/instructions/shared/untrusted-content-boundary.md # docs/reference/prompts/README.md # docs/reference/prompts/data-science-engineering/synth-data-generate.md # docs/reference/skills/README.md # docs/reference/skills/data-science-engineering/analysis-authoring.md # docs/reference/skills/data-science-engineering/data-catalog.md # docs/reference/skills/data-science-engineering/data-science-engineering-foundation.md # docs/reference/skills/data-science-engineering/dataops.md # docs/reference/skills/data-science-engineering/evaluation-design.md # docs/reference/skills/data-science-engineering/feasibility.md # docs/reference/skills/data-science-engineering/ml-experimentation.md # scripts/tests/lib/MarketplaceClosure.Tests.ps1
…ter merge Main's single-plugin consolidation retired the per-package plugin documentation pages and the marketplace test suite. Both survived the merge as modify/delete conflicts resolved toward this branch, breaking frontmatter schema validation and the PowerShell test lane.
Katrien De Graeve (katriendg)
approved these changes
Aug 18, 2026
Katrien De Graeve (katriendg)
left a comment
Katrien De Graeve (katriendg)
left a comment
Contributor
There was a problem hiding this comment.
Went through this one in depth — agents, skills, docs, evals, and the generated surfaces — and ran the local validation lanes against the branch.
Approving. Both consolidations are architecturally right. Folding the supply-chain pipeline into SSSC Reviewer removes a genuinely duplicated orchestrator and keeps report authorship local instead of routing through Report Generator's incompatible template. Hoisting privacy, RAI, and telemetry into trigger-based cross-cutting concerns is a better model than the per-job table it replaces. Nothing below asks you to rethink either one — they are absorption-hygiene and rename-sweep gaps, all local text edits. Approving rather than blocking because you reliably close findings before merging.
I left seven inline comments — one High, five Medium, one Low.
Nice piece of work — the git mv-separate-from-rewrite discipline is what made 112 files reviewable.
Reviewed with AI assistance; every finding was independently verified against the diff and the referenced files before filing. Treat it as input to human review, not a substitute for it.
Open
9 tasks
…g-consolidation # Conflicts: # .github/plugin.json # docs/reference/skills/README.md
…g-consolidation # Conflicts: # docs/reference/skills/README.md
Jamie Kim (jkim323)
approved these changes
Aug 21, 2026
- complete SSSC Reviewer retry protocol and status/verdict translation - emit the canonical SSSC Planning disclaimer verbatim - drop coach copies of job-registry cross-cutting rules - correct agent-behavior counts to 56 agents, 30 parents, 26 subagents - fix last stale ds-catalog reference 🔧 - Generated by Copilot
….com/microsoft/hve-core into refactor/agent-catalog-consolidation
- add degraded merge-base fallback when pr-reference is unavailable - reconcile reviewer guide inputs, prerequisites, and quick start - add isolated diff-mode regression stimulus and regenerate eval spec 🔒 - Generated by Copilot
The SSSC agent-behavior lane failed intermittently because its graders asserted output the evaluation sandbox cannot produce. The class-recipe stimulus rewarded invented findings and penalized honest reports that evidence was unavailable, and the diff stimulus required a claim of direct repository comparison that a sandbox without a repository cannot support. Graders are now derived from measured responses across repeated local runs. The class-recipe stimulus checks the report path, supply-chain framework scope, a completion statement, self-consistency, and absence of source edits. The diff stimulus accepts an honest changed-file resolution basis and a collective file-retention statement. Several patterns used a negated-period bound to span context, which cannot cross a sentence boundary and silently failed on correct output. Those bounds are corrected, and the no-source-edit guard is aligned with the reviewed RV-001 fix so a .json path no longer matches the .js alternative. Validated by re-grading frozen transcripts offline and by five independent local runs: mean 97.3 percent, minimum 93.3 percent, no errored trials. A seventeen-case adversarial matrix confirms each regression still fails its designated grader.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates an overlapping corner of the agent catalog and aligns the data-science package vocabulary. Three related changes travel together; they share no files and are separable by commit.
Supply Chain ReviewerandSSSC Reviewerboth assessed the singlesupply-chain-securityskill across the same audit, diff, and plan modes.SSSC Revieweris the load-bearing one — two prompts, thevex-draftworkflow, an instructionsapplyToglob, thevexskill, avex-detectuser message, the documented security model, and a PowerShell eval fixture all bind to it.Supply Chain Reviewerhad none of those bindings.data-science-engineering, the coach becomes the Data Science and Engineering Coach, and five skills drop theds-prefix.Dependency Reviewerwas evaluated and deliberately left standalone.Why the duplicate wasn't just deleted
Supply Chain Reviewerwas the only agent that actually dispatched the assessment pipeline.SSSC Reviewerdeclared four subagents in frontmatter but assessed inline and never called them. A plain delete would have discarded adversarial verification of FAIL/PARTIAL findings and orphanedsupply-chain-skill-assessor.agent.md, whose sole dispatcher was the file being removed.So the pipeline was ported first (
Codebase Profiler→Supply Chain Skill Assessor→Finding Deep Verifier), then the duplicate was removed.Report Generatorwas deliberately excluded: its report roots are hard-coded to.copilot-tracking/security, so adopting it would have reintroduced the exact report-root collision this consolidation removes.SSSC Reviewercontinues authoring its own nine-section report into.copilot-tracking/sssc-reviews/.Things reviewers should know
Supply Chain ReviewersharedSecurity Reviewer's report namespace — identical directory, filename pattern, and sequence counter. Its removal resolves that collision as a side effect.marketplace.jsonalready shippedprivacy-standards,rai-standards,rai-planner, andtelemetry-foundationsin this package, and the job registry already named three of them. But the coach's stop rules said to "stop rather than cross a seven-skill boundary" — contradicting the registry it is told to load.rai-standardswas routed nowhere at all.project-planningshippedSSSC Reviewerwithout the subagents it now dispatches. Marketplace closure keys onhandoffs, not theagentslist, soplugin:validatewould not have caught this. The package now carries the three subagents plussecurity-reviewer-formats.data-science-engineering/ HVE Core - Data Science and Engineering.Deliberately preserved
DS_CATALOG_V1, its schema$id, and its schema/template filenameshve-core/architecture-diagramsreads it, and users may have authored catalogs with itdata-workstream-session-v1state schema version.copilot-tracking/ds/session root.copilot-tracking/dt/convention; renaming would silently orphan in-flight user sessionsRename history
Every rename is a committed
git mvseparate from its content rewrite. Because this repo squashes PRs, branch commits do not reachmainand only final-tree similarity determines rename detection — so verification is a merge-base check rather than a branch-onlygit log --follow:Validation
lint:plugin-output,lint:tables,lint:md,lint:ps,lint:yaml,lint:jsonlint:frontmatter(897 files)plugin:validate(10 plugins)lint:hooks,lint:version-consistency(627 pinned actions)lint:py(16 skills),validate:skills(72 skills)lint:ai-artifacts,lint:modelslint:asset-docsdocs:generate:check(235 pages)lint:links,lint:md-linksTwo issues surfaced during validation and were fixed: an initial package
versionbump to4.0.0was rejected because package versions are lockstep with rootpackage.json, and table formatting drifted across ten files (format:tablesapplied).Generated output
docs/reference/was regenerated withdocs:generate; thedocs/plugins/marker region belongs to a different generator (plugin:generate) and was refreshed with an external staging root. The generator preserved eight renamed reference pages as orphans because they carried authored tails — those tails were carried forward before the orphans were removed, so no hand-written "When to use it" content was lost.Review notes
hve-builderreview gate ran against the coach rewrite and returned Revise; all six findings were applied, including one real inconsistency where the RAI trigger differed between the agent and the registry.Related issues
Closes #2733.
SSSC Reviewerdiff mode now resolves its changed-files list from a defined producer, including a degraded path that stays indiffmode whenpr-referencecannot be activated, the reviewer documentation matches that contract, and an isolated diff-mode stimulus locks the behavior.🤖 Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.