feat(agents): Add DS Evaluation Strategist for AI and data science evaluation methodology design

[Anshuman5220]

Custom Agent Name

/ds-evaluation-strategist

Type

Custom Agent

Purpose

Core mission: The DS Evaluation Strategist takes any AI project description — a PRD, a problem statement, or even a single sentence — and produces a structured evaluation methodology plan (eval_methodology_{version}.md). The plan defines what to measure, why each metric matters, what evidence is needed, and what decision each result supports. It tracks every claim as confirmed, inferred, or assumed, and progresses through readiness stages (Draft → Zero-Shot Baseline → Ready) so teams always know how far they are from a defensible evaluation. The goal is to answer "how do we know this works?" before implementation begins, not after.

Why it should exist in HVE Core: hve-core has agents for research, planning, implementation, and review, but nothing for evaluation methodology design. Teams re-derive the same evaluation decisions from scratch on each project, often inconsistently. This is a recurring gap across AI and data science work.

When to use it: After planning artifacts exist and before implementation begins. A team has a problem statement (any completeness level) and needs to turn it into a defensible evaluation methodology. Works well under ambiguity, where requirements are incomplete but teams still need to move forward with clearly marked assumptions.

Why a custom agent, not a prompt or instruction: Evaluation methodology design needs persistent expertise — a multi-phase workflow with gap routing, domain modules, metric derivation reasoning, assumption tracking, and iterative refinement. A prompt cannot maintain this state. An instruction file cannot drive a multi-turn workflow with peer checkpoints and circuit breakers.

Feedback ask: I'm proposing this as an initial design for feedback rather than assuming the final shape. I would appreciate maintainer guidance on whether this belongs in the Data Science collection, whether /ds-evaluation-strategist is the right name and scope, and whether this should be standalone or integrated into an existing workflow. I have a working prototype of DSEvaluationStrategist.agent.md and a companion README.md, and I'm happy to share through a PR branch or revise based on feedback.

DSEvaluationStrategist.agent.md
README.md

Requirements

Persona

The agent acts as an evaluation methodology specialist: an expert in statistical design, metric derivation, and evaluation planning across AI and data science problem types. It operates as a collaborative advisor to data scientists by surfacing the right questions and structuring the methodology, while leaving final judgment to humans.

Inputs

The agent should accept:

• PRDs, problem statements, and system descriptions at any completeness level, from a single sentence to a full specification
• Follow-up answers to clarifying questions
• Delegated task results returned from team members, such as data profiles, label samples, SLA numbers, or domain constraints

Workflow process

The agent should support the following workflow:

1. Discovery and gap analysis

   • Ingest the input.
   • Classify the problem type.
   • Extract core assumptions such as baseline, risk, and business value.
   • Identify missing information.
   • Load relevant domain modules as reference.

2. Task delegation

   • Route unresolved gaps to appropriate roles such as Product Owner, Domain SME, Data Engineer, or Platform Engineer.
   • Include context on why each missing input matters.
   • Use a circuit breaker after maximum clarification iterations.

3. Peer checkpoint

   • Present assumed classification, baseline, top risks, and proposed evaluation direction.
   • Ask for confirmation before investing in full methodology generation.

4. Framework generation

   • Produce the full evaluation methodology anchored by evidence, derived metrics, decision rules, assumptions, limitations, and unresolved gaps.

Expected outputs

The agent should produce a versioned eval_methodology_{version}.md document containing:

• Readiness status: (🔴 Draft → 🟡 Zero-Shot Baseline → 🟢 Ready)
• Provenance on all claims: (confirmed / inferred / assumed ⚠️)
• Outer Loop metrics for system-level evidence, such as business value, user trust, and operational health
• Inner Loop metrics for component-level correctness
• Metrics derived through reasoning: objective -> decision -> failure mode -> metric
• Golden dataset strategy
• Error taxonomy
• Evaluation stages: offline -> pilot -> production
• Reporting gates
• Statistical rigor guidance, including confidence intervals, minimum detectable effect, and decision rules where applicable
• Task delegation backlog for unresolved gaps
• Domain module references for Retrieval and RAG, Generative LLM, Classification, Agents, Recommendation, Forecasting, A/B Testing, Fairness, and Calibration

Quality standards

The agent output should meet the following standards:

• Metrics are recommended only after explaining what decision they support and what failure mode they detect.
• Checklist-only metric selection is not sufficient; derivation reasoning is required.
• A peer checkpoint is required before full generation to reduce wasted effort caused by incorrect assumptions.
• An escalation report is produced after maximum clarification iterations.
• The methodology document is versioned and incremented on major structural changes.
• The methodology document acts as the primary working artifact and session state.
• Outputs should be structured, reproducible, reviewable, assumption-aware, statistically grounded, and suitable for human review.

Boundaries / non-goals

The agent must maintain an evaluation-only boundary:

• Does not recommend architecture or implementation approach
• Does not select models or vendors
• Does not implement code
• Does not execute evaluation runs
• Does not replace human judgment or stakeholder sign-off
• Does not generate production readiness approval

[katriendg]

Thanks for the thoughtful proposal, Anshuman Singh Bhadauria (@Anshuman5220)! This is a genuine gap in hve-core and we'd be very interested in a PR. The agent-as-living-document pattern, the provenance markers, and the metric derivation (vs. checklist) approach all align well with how rai-planner and security-planner are structured.

A few notes to guide the PR before you open it.

Collection placement

Please target the Data Science collection. It sits naturally next to eval-dataset-creator, gen-data-spec, and rai-planner. That means updates in the same PR to:

• .github/agents/data-science/<filename>.agent.md (kebab-case filename matching the slash command)
• collections/data-science.collection.yml
• collections/hve-core-all.collection.yml
• .github/copilot-instructions.md — add the new tracking directory (we suggest .copilot-tracking/evaluations/, plural to match security-plans, rai-plans, dt, etc.) under the Copilot Tracking inventory
• Run npm run plugin:generate and npm run lint:all before pushing

Please follow the contributing guides:

• docs/contributing/README.md
• docs/contributing/custom-agents.md
• docs/contributing/ai-artifacts-common.md

We also recommend running your draft through the Prompt Builder agent and the Prompt Tester agent before submission. Prompt Builder will help you align frontmatter (argumentHint casing, tools declaration), file/heading conventions, and writing-style rules; Prompt Tester will surface ambiguities by executing the agent literally.

Two small adjustments to the draft we'd ask for up front:

• Drop the /compact self-handoff. Either omit handoffs or hand off to the Memory agent for session continuity (consistent with other planning agents in the repo).
• Mark the agent maturity: experimental in the collection YAML initially, mirroring rai-planner.

Overlap with eval-dataset-creator — please research and reconcile

This is the area we'd like you to dig into deepest before the PR. We did a preliminary side-by-side and found partial overlap on different lifecycle stages, not duplication — but there is real risk of user confusion because both names contain "eval" and both live under data-science/. Summary of what we found:

Dimension	eval-dataset-creator (existing)	Your proposal
Lifecycle stage	Just-before-execution: dataset + tooling artifacts	Pre-implementation: methodology plan
Output	{agent}-eval-dataset.json/.csv + curation/metrics/tools docs	eval_methodology_{version}.md
Audience	Citizen Dev (Copilot Studio) and Pro-Code (Azure AI Foundry)	Data scientists / ML engineers, tooling-agnostic
System scope	Conversational AI agents (Q&A, grounding, tool-calling)	Any AI/ML system: classification, RAG, recommendation, forecasting, agents, generative
Metric approach	Catalog-based selection (Intent Resolution, Groundedness, Tool Call, etc.)	Derivation: objective to decision to failure mode to metric
Statistical rigor, gates, error taxonomy, golden-set strategy across modalities	Not addressed	Core focus
Dataset construction (>=30 pairs, balance)	Core focus	Out of scope (delegates)

Our recommended approach for the two agents:

1. Position by output and audience, not the word "evaluation." Sharpen both descriptions so the choice is obvious at a glance — e.g. "Generate evaluation datasets for chat agents on Copilot Studio / AI Foundry" vs. "Design an evaluation methodology plan for any AI/ML system, before dataset creation."
2. Make the handoff explicit in both directions. Your agent should state: "If the methodology selects a chat/Copilot Studio/Foundry agent and metrics are agreed, hand off to eval-dataset-creator for dataset construction." We will mirror a "for methodology design, use the strategist first" note in eval-dataset-creator.
3. Consider naming for disambiguation. We'd like your view on whether eval-methodology-planner (or a similar output-anchored name) is clearer than ds-evaluation-strategist — the repo's existing pattern (gen-data-spec, gen-jupyter-notebook, gen-streamlit-dashboard) names by output, and ds- is somewhat redundant inside the data-science collection. Happy with whatever you propose as long as the rationale is in the PR description.
4. Avoid re-deriving Foundry/MCS metrics from scratch. When your "Agents & Workflows" lens applies, reference eval-dataset-creator's metric catalog rather than independently re-derive equivalent metrics — keeps the agents reinforcing instead of competing.
5. Defer fairness/bias and RAI control surfaces to rai-planner. Reference rather than reproduce. However, we do not suggest to take a full dependency on RAI agent or instructions, as the Data Science collection does not package these, and we want to try and keep cross-collection linking to minimum (as it requires duplicating agents if users install multiple collections).
6. Add a "When to use which" decision aid to collections/data-science.collection.md (or to both agent descriptions) so users can self-route quickly.

Please research these in depth in the PR description: where the genuine overlap is, what each agent uniquely contributes, and how a user is supposed to choose. We want to take the best of both and end up with two clearly differentiated, complementary agents.

Ask

Please open an initial PR with:

• The agent file in the right location with corrected frontmatter
• Collection YAML/MD updates
• copilot-instructions.md tracking directory addition
• The overlap research and naming proposal in the PR description
• A short disclaimer in the agent itself noting outputs are assistive and require qualified DS / stakeholder review (see disclaimer-language.instructions.md)
• Document this agent in the docs folder, and in CUSTOM-AGENTS.md

We likely have additional suggestions (template trims, integration points, response formatting), but we'd rather give them on a concrete PR for a thorough review than front-load everything here. Excited to see this land — thanks again for the careful design work!

[katriendg]

Anshuman Singh Bhadauria (@Anshuman5220) I already assigned this issue to you so no one picks it up and you can take your time working on it. Thanks again.

[Anshuman5220]

Katrien De Graeve (@katriendg) - Thanks for the detailed analysis and review. I really appreciate the depth of the feedback, especially the side-by-side comparison with eval-dataset-creator and the naming suggestion.

Why both agents should coexist

My read is that the two agents solve related but different problems. eval-methodology-planner is useful whether or not a team already has evaluation data. Even if a team has 1,000 labeled Q&A pairs, they still need to decide which metrics matter, what decision each result supports, what the minimum detectable effect should be, and what action to take when a metric fails. That is methodology work, and it is independent of dataset availability.

eval-dataset-creator is more conditionally relevant when a team needs to construct a chat-agent test dataset from scratch, especially for Copilot Studio or Foundry scenarios. So the handoff from eval-methodology-planner to eval-dataset-creator is one possible path, but not the only path.

Overlap with eval-dataset-creator

I reviewed this carefully, and I agree with your analysis that these are complementary lifecycle stages. eval-methodology-planner answers: “What should we measure, why should we measure it, and what counts as evidence?” across AI/ML system types. eval-dataset-creator answers: “What are the test cases and tooling needed to evaluate this?” for conversational AI agents specifically.

The main overlap risk is in the Agents & Workflows lens. eval-methodology-planner could derive metrics like task completion or tool-use accuracy using different names from eval-dataset-creator’s catalog, such as Intent Resolution, Task Adherence, and Tool Call Accuracy. That would create unnecessary confusion during handoff. I’ll address this by having eval-methodology-planner reference eval-dataset-creator’s catalog terms for Copilot Studio and Foundry agents, so both agents use the same metric language.

For fairness and bias, I’ll reference rai-planner without creating a hard dependency. I’ll also add explicit handoffs in both directions, a “when to use which” decision aid, and the mirror note in eval-dataset-creator.

Naming

I like your suggestion of eval-methodology-planner. It names the output, follows the repo convention used by agents like gen-data-spec and gen-jupyter-notebook, removes the redundant ds- prefix, and clearly differentiates it from eval-dataset-creator. Happy to go with it.

Structural items

All confirmed. I’ll handle the data-science collection placement, remove /compact, mark the agent as experimental, add the disclaimer from disclaimer-language.instructions.md, run Prompt Builder/Tester and lint, document it in docs/ and CUSTOM-AGENTS.md, and follow the contributing guides.

Next step

As suggested, I’ll open a draft PR with the full analysis in the description after working through the items above. Thanks again for the thoughtful and actionable review.

[katriendg]

Fantastic, thanks for taking this one and looking forward to your PR Anshuman Singh Bhadauria (@Anshuman5220)

[Anshuman5220]

Hi
I will resume work on this , as i was out of office for some time

[WilliamBerryiii]

Anshuman Srivastava (@anshuman) Singh Bhadauria (Anshuman5220) - can you hold on this for a bit. We are really trying to decrease the number of agents and map more of the capability into skills that a fewer number of agents utilized through a phased based protocol.

I'm currently reworking the DS agent to add similar functionality from both a coaching and planning perspective. Let me get my PR up and I'd like to ask you to review it and help me identify gaps.

[anshuman]

Anshuman Singh Bhadauria (@Anshuman5220) I believe the comment above is for you.