Skip to content

fix(agents): align RAI assessments with supplied materials - #2764

Open
okada-kai wants to merge 4 commits into
microsoft:mainfrom
okada-kai:fix/rai-template-population
Open

fix(agents): align RAI assessments with supplied materials#2764
okada-kai wants to merge 4 commits into
microsoft:mainfrom
okada-kai:fix/rai-template-population

Conversation

@okada-kai

Copy link
Copy Markdown

Pull Request

Description

Problems and fixes

  • Late input discovery: Assessments could start before confirming available
    project materials and optional templates, causing rework or missing evidence.
    Fix: Phase 1 checks optional templates before project materials and
    permitted evidence sources.
  • Incomplete template output: Sample rows or widgets could limit generated
    content to only part of the assessment. Fix: Supplied templates use a
    stable-ID assessment-content.md projection, and rows and widgets expand
    beyond existing placeholders.
  • Unprofessional report style: Reports could include conversational,
    drafting, or tool narration that required cleanup. Fix: Generated
    documents use assessment-focused prose and exclude process narration.

Related Issue(s)

None.

Type of Change

Select all that apply:

Code & Documentation:

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update

Infrastructure & Configuration:

  • GitHub Actions workflow
  • Linting configuration (markdown, PowerShell, etc.)
  • Security configuration
  • DevContainer configuration
  • Dependency update

AI Artifacts:

  • Reviewed contribution with hve-builder and addressed all actionable findings
  • Copilot instructions (.github/instructions/*.instructions.md)
  • Copilot prompt (.github/prompts/*.prompt.md)
  • Copilot agent (.github/agents/*.agent.md)
  • Copilot skill (.github/skills/*/SKILL.md)
  • Copilot hook (.github/hooks/*/*.json)
  • Eval spec added/updated for changed AI artifacts (evals/)

Note for AI Artifact Contributors:

  • Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review .github/agents/ before creating new ones.
  • Skills: Must include both bash and PowerShell scripts. See Skills.
  • Model Versions: Contributions MUST target models listed in the model catalog (scripts/linting/model-catalog.json) whose provider appears in providerAllowlist and whose status is ga or preview. Run npm run lint:models to validate references.
  • See Agents Not Accepted and Model Version Requirements.

Other:

  • Script/automation (.ps1, .sh, .py)
  • Other (please describe):

Sample Prompts (for AI Artifact Contributions)

User Request:

Create an RAI assessment from project evidence and populate the supplied
impact-assessment document and Mural templates.

Execution Flow:

  1. Uses the existing entry mode and six-phase workflow.
  2. Resolves the project slug and output requirements, then checks all optional
    document and Mural templates before project materials.
  3. Persists each supplied template reference and creates the required
    assessment-content.md skeleton.
  4. Populates the projection from project evidence without limiting repeatable
    rows or Mural widgets to existing placeholders.
  5. Records authoritative findings in rai-plan.md and prepares only the
    outputs produced by the assessment.

Output Artifacts:

  • .copilot-tracking/rai-plans/{project-slug}/rai-plan.md containing the
    canonical assessment findings, evidence, controls, decisions, and tradeoffs.
  • .copilot-tracking/rai-plans/{project-slug}/assessment-content.md
    containing stable rendering IDs and the combined structure of every supplied
    template.
  • Populated document and Mural outputs when requested.
  • Reviewed and sanitized backlog handoff artifacts after confirmation.

Success Indicators:

  • All six phases execute sequentially with the documented hard gates.
  • Phase 1 checks templates before project materials and WorkIQ evidence.
  • Assessment conclusions originate in rai-plan.md.
  • Missing templates cannot block the workflow; when templates are supplied,
    assessment-content.md is required before they are populated.
  • Placeholder counts do not limit repeatable rows, widgets, or sticky notes.
    Existing Mural widgets and anchors define layout only; the planner creates
    one widget for every stable-ID content row.
  • Without a Mural template, A1 derives from ## System Definition, A2 from
    ### AI Component Inventory, and A3 from ## Stakeholder Impact.
  • Recovery preserves consent history and never infers hard-gate confirmation.
  • Final documents contain assessment prose rather than drafting or tool
    narration.

For detailed contribution requirements, see:

Testing

All validation ran under Node.js 24:

  • npm run test:ps -- -TestPath scripts/tests/linting/Test-RaiStateSchema.Tests.ps1
    passed 7 of 7 tests.
  • npm run docs:generate:check
  • npm run plugin:validate
  • npm run validate:docs
  • npm run spell-check
  • npm run lint:md-links
  • npm run lint:tables
  • npm run lint:frontmatter
  • npm run lint:ai-artifacts
  • npm run validate:skills
  • npm run validate:local

Diff review confirmed that Entry Modes and suggestedDepthTier behavior remain
unchanged from the base branch and that placeholder/cardinality behavior is
contained in one commit. The HVE Builder behavior simulation passed with no
in-scope findings. Native Mural, WorkIQ, backlog, signing, and file-write
behavior was not exercised.

Checklist

Required Checks

  • Documentation is updated
  • Files follow existing naming conventions
  • Changes are backwards compatible
  • Tests added for new functionality

The six-phase workflow and Entry Modes remain unchanged. The state schema now
requires persisted template-preflight records, so state created before this
change requires migration.

AI Artifact Contributions

  • Used hve-builder review mode to review contribution
  • Addressed all actionable findings from the hve-builder review
  • Verified contribution follows common standards and type-specific requirements

Required Local Checks

The following local-safe validation commands must pass before merging:

  • Local validation aggregate: npm run validate:local
  • Documentation validation (if docs changed): npm run validate:docs
  • Spell checking: npm run spell-check
  • Link validation: npm run lint:md-links

Security Considerations

  • This PR does not contain any sensitive or NDA information
  • Any new dependencies have been reviewed for security issues (N/A — no dependency changes)
  • Security-related scripts follow the principle of least privilege (N/A — no security script changes)

GHCP Artifact Maturity

Warning

This PR includes an experimental GHCP artifact that may have breaking changes.

  • .github/agents/rai-planning/rai-planner.agent.md
  • .github/instructions/rai-planning/rai-identity.instructions.md
File Type Maturity Notes
.github/agents/rai-planning/rai-planner.agent.md Agent ⚠️ experimental Pre-release only
.github/instructions/rai-planning/rai-identity.instructions.md Instruction ⚠️ experimental Pre-release only

GHCP Maturity Acknowledgment

  • I acknowledge this PR includes non-stable GHCP artifacts
  • Non-stable artifacts are intentional for this change

Additional Notes

The repository pull request template was preserved. Required AI artifact and
maturity acknowledgments are complete. Nothing has been pushed and no pull
request has been created.

okada-kai and others added 3 commits August 24, 2026 14:42
- route every entry mode through template-first preflight
- persist template state for resume and recovery
- validate preflight state fixtures

🔍 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- create one Mural widget for every supported content row
- preserve rai-plan sources when no template is supplied
- allow impact assessment structures to grow with evidence

🧩 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- use neutral assessment-focused report prose
- exclude conversation and drafting narration from reports

📝 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@okada-kai okada-kai changed the title fix(agents): prevent incomplete RAI template output fix(agents): align RAI assessments with supplied materials Aug 24, 2026
@okada-kai

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="Microsoft"

@okada-kai
okada-kai marked this pull request as ready for review August 26, 2026 01:29
@okada-kai
okada-kai requested a review from a team as a code owner August 26, 2026 01:29
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.07%. Comparing base (3c3dc02) to head (eda4136).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2764      +/-   ##
==========================================
+ Coverage   82.98%   83.07%   +0.08%     
==========================================
  Files         183      171      -12     
  Lines       33787    33947     +160     
  Branches       25        0      -25     
==========================================
+ Hits        28038    28200     +162     
- Misses       5746     5747       +1     
+ Partials        3        0       -3     
Flag Coverage Δ
docusaurus ?
pester 83.83% <ø> (+0.37%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 15 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants