diff --git a/.github/agents/data-science-engineering/data-science-engineering-coach.agent.md b/.github/agents/data-science-engineering/data-science-engineering-coach.agent.md index 108ade2b28..ec20de4875 100644 --- a/.github/agents/data-science-engineering/data-science-engineering-coach.agent.md +++ b/.github/agents/data-science-engineering/data-science-engineering-coach.agent.md @@ -29,7 +29,8 @@ durable write. `feasibility` owns evidence-led studies and interchange traceability; `analysis-authoring` owns notebook and dashboard composition and dashboard validation; `evaluation-design` owns AI-system evaluation dataset design; - `experiment-design` owns general experiment framing and evaluation; and + `experiment-design` owns problem-class framing for a stated business outcome + as well as general experiment framing and evaluation; and `ml-experimentation` owns ML-specific reproducibility, tracking, evaluation, abstractions, and readiness. * `privacy-standards`, `rai-standards`, `rai-planner`, and @@ -110,8 +111,10 @@ route: visualization selection, and dashboard validation. * `evaluation-design`: AI-system evaluation dataset design, difficulty balance, metric selection, and evaluation tooling fit. -* `experiment-design`: general experiment selection, hypotheses, vetting, - minimum scope, and result interpretation. +* `experiment-design`: problem-class framing that turns a stated business + outcome into candidate data-science problem classes without selecting one, + and general experiment selection, hypotheses, vetting, minimum scope, and + result interpretation. * `ml-experimentation`: ML environments, reproducibility, tracking, evaluation, dataset and model abstractions, and production readiness. diff --git a/.github/instructions/hve-core/licensing-posture.instructions.md b/.github/instructions/hve-core/licensing-posture.instructions.md index eea97007da..8846fcf65d 100644 --- a/.github/instructions/hve-core/licensing-posture.instructions.md +++ b/.github/instructions/hve-core/licensing-posture.instructions.md @@ -77,9 +77,15 @@ Requirements for any verbatim or lightly edited reproduction: ### Mixed-content packages -A skill package that combines third-party-derived content with repository-original content declares a compound SPDX expression naming both, for example `MIT AND CC-BY-4.0`. Its attribution section states which files belong to which license. Declaring only one license misrepresents the other body of content. +A skill package that combines third-party-derived content with repository-original content declares a compound SPDX expression naming every license present in the package, for example `Apache-2.0 AND CC-BY-4.0`, and allocates licenses per file in its attribution section. Declaring only one license misrepresents the other body of content. Never declare a non-SPDX placeholder such as `mixed`; the frontmatter field is an SPDX expression and a placeholder tells a consumer nothing. -When both bodies carry the same license, declare that single license rather than a degenerate compound expression. The attribution section still identifies which content is third-party-derived: by file where the split is by file, and inline where derived and original material share a file. +At package level, `AND` is the conjunction a redistributor faces: whoever redistributes the whole package complies with every named license. That is the correct reading for a mixed package. It is not a claim that every file carries every license, which is what the per-file allocation table exists to state. Use `OR` only for a genuine recipient choice. + +A license that imposes no obligation adds no term. Public-domain source material, including U.S. Government works, and paraphrase-only use of open legal text produce repository-original expression, so neither appears in the expression. A derivative of a copyleft source does appear, because ShareAlike propagates. + +Allocate by material class, following the per-file practice used by the [Linux kernel licensing rules](https://docs.kernel.org/process/license-rules.html) and the documentation-versus-code split published by [Kubernetes](https://kubernetes.io/docs/contribute/style/style-guide/). The attribution section names each file or directory and the license that governs it. Where derived and original material share one file, mark the boundary inline. + +When every body of content carries the same license, declare that single license rather than a degenerate compound expression. The attribution section still identifies which content is third-party-derived. ### Open legal text (statutes and regulations) @@ -105,7 +111,7 @@ Verbatim restricted-standard text is a licensing violation and is reverted at re * Do not reproduce an entire upstream page, section, or document regardless of license. Treat a long or substantial excerpt as a gating license-risk finding, not an advisory one. * Verbatim text is forbidden for restricted standards (ISO, IEC, ETSI) under any circumstance, including short partial quotes, table rows, and figure captions. * A derivative of a CC BY-SA source carries the ShareAlike notice and its source attribution into this repository. -* A skill package whose reference content spans more than one license declares `license: mixed` in its frontmatter rather than a single identifier that covers only part of the package. +* A skill package whose reference content spans more than one license declares a compound SPDX expression naming every license present, plus a per-file allocation table, rather than a single identifier that covers only part of the package or a non-SPDX placeholder such as `mixed`. * When the licensing posture for a specific snippet is ambiguous, paraphrase rather than quote. * Preserve standards identifiers verbatim (clause numbers, control IDs, criterion IDs); identifiers are facts, not licensed prose. * Treat long or substantial excerpts as a license-risk finding during review. diff --git a/.github/skills/data-science-engineering/data-science-engineering-foundation/references/job-registry.md b/.github/skills/data-science-engineering/data-science-engineering-foundation/references/job-registry.md index 34021e0d5f..d93aad6776 100644 --- a/.github/skills/data-science-engineering/data-science-engineering-foundation/references/job-registry.md +++ b/.github/skills/data-science-engineering/data-science-engineering-foundation/references/job-registry.md @@ -13,17 +13,18 @@ it does not duplicate the methods owned by a skill. ## Registry -| Job | Class | Primary route | Optional supporting route | Durable output or completion evidence | -|-----------------|--------------|------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------| -| `catalog` | `continuous` | `data-catalog`: durable catalog entities, declared relationships, lineage, and model semantics | `analysis-authoring` when a presented dataset profile is the requested output | Caller-approved data catalog | -| `model-diagram` | `episodic` | `data-catalog`: authoritative declared entities and relationships | `architecture-diagrams`: render the declared model without becoming semantic authority | Mermaid or ASCII diagram | -| `feasibility` | `bounded` | `feasibility`: evidence-led feasibility studies with durable traceability | None | Caller-approved feasibility study | -| `pipeline` | `episodic` | `dataops`: tier behavior, pipeline invariants, validation, tests, and drift | None | Transformation, validation, or pipeline code | -| `analysis` | `episodic` | `analysis-authoring`: EDA notebook and analytical dashboard authoring and validation | `data-catalog` for column semantics; `dataops` for derived-dataset persistence | Notebook, dashboard, or analysis deliverable | -| `evaluation` | `episodic` | `evaluation-design`: AI-system evaluation dataset design, metrics, and tooling | None | Evaluation dataset with curation, metric, and tooling documents | -| `experiment` | `episodic` | `experiment-design`: experiment selection, hypotheses, vetting, scope, and evaluation | `ml-experimentation`: ML reproducibility, tracking, evaluation, abstractions, and production readiness | Hypothesis, experiment assets, and result disposition | -| `testing` | `episodic` | `dataops`: DataOps and DS/MLOps test techniques | `analysis-authoring` for dashboard validation technique | Test code and assertions | -| `observability` | `episodic` | `dataops`: data/model signals and validation-versus-drift guidance | None | Instrumentation code and signal recommendations | +| Job | Class | Primary route | Optional supporting route | Durable output or completion evidence | +|-------------------|--------------|----------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------| +| `catalog` | `continuous` | `data-catalog`: durable catalog entities, declared relationships, lineage, and model semantics | `analysis-authoring` when a presented dataset profile is the requested output | Caller-approved data catalog | +| `model-diagram` | `episodic` | `data-catalog`: authoritative declared entities and relationships | `architecture-diagrams`: render the declared model without becoming semantic authority | Mermaid or ASCII diagram | +| `problem-framing` | `episodic` | `experiment-design`: candidate data-science problem classes for a stated business outcome, with entry-test reasoning | None | Candidate framing set with reasoning, unknowns, and gaps | +| `feasibility` | `bounded` | `feasibility`: evidence-led feasibility studies with durable traceability | None | Caller-approved feasibility study | +| `pipeline` | `episodic` | `dataops`: tier behavior, pipeline invariants, validation, tests, and drift | None | Transformation, validation, or pipeline code | +| `analysis` | `episodic` | `analysis-authoring`: EDA notebook and analytical dashboard authoring and validation | `data-catalog` for column semantics; `dataops` for derived-dataset persistence | Notebook, dashboard, or analysis deliverable | +| `evaluation` | `episodic` | `evaluation-design`: AI-system evaluation dataset design, metrics, and tooling | None | Evaluation dataset with curation, metric, and tooling documents | +| `experiment` | `episodic` | `experiment-design`: experiment selection, hypotheses, vetting, scope, and evaluation | `ml-experimentation`: ML reproducibility, tracking, evaluation, abstractions, and production readiness | Hypothesis, experiment assets, and result disposition | +| `testing` | `episodic` | `dataops`: DataOps and DS/MLOps test techniques | `analysis-authoring` for dashboard validation technique | Test code and assertions | +| `observability` | `episodic` | `dataops`: data/model signals and validation-versus-drift guidance | None | Instrumentation code and signal recommendations | The cross-cutting concerns below are evaluated in addition to the routes in this table. They are not listed per job, because they apply by trigger rather than by @@ -58,19 +59,19 @@ The Data Science and Engineering Coach routes to the data science and engineering skills below, plus the cross-cutting skills above. Keep their authority separate even when one job loads more than one skill. -| Exact skill name | Capability description | Does not own | -|-------------------------|-----------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------| -| `data-catalog` | Durable catalog entities, declared relationships, identity, coverage, lineage, and attached dataset profiles | Pipeline tier behavior, inferred diagram relationships, or feasibility verdicts | -| `dataops` | DataOps tier semantics, pipeline invariants, validation, tests, drift, signals, and derived-dataset persistence | Catalog meaning, experiment selection, ML tracking setup, or sensitivity classification | -| `feasibility` | Evidence-led feasibility studies, lifecycle, recommendations, and interchange traceability | Functional requirement numbering, downstream writeback, catalog semantics, or implementation | -| `analysis-authoring` | EDA notebook and analytical dashboard composition, visualization selection, and dashboard validation | Column semantics, persistence format, accessibility conformance, or evaluation dataset design | -| `evaluation-design` | AI-system evaluation dataset design, difficulty balance, metric selection, and tooling fit | Trained-model evaluation, Responsible AI approval, or session and durable-write mechanics | -| `experiment-design` | Experiment selection, hypotheses, vetting, minimum scope, and result evaluation | ML infrastructure, production implementation, or pipeline mechanics | -| `ml-experimentation` | ML environments, reproducibility, tracking, abstractions, evaluation, and readiness | General hypothesis framing, pipeline replay, or Responsible AI approval | -| `privacy-standards` | Sensitivity classification, data-flow reasoning, standards mapping, and DPIA thresholds | Catalog semantics, pipeline mechanics, or durable-write authority | -| `rai-standards` | Responsible AI standards, risk framing, and standards mapping | Evaluation dataset design, model performance, or approval authority | -| `rai-planner` | Scoped Responsible AI assessment when a surfaced risk warrants one | Data science method authority, session mechanics, or approval authority | -| `telemetry-foundations` | Metric names, instruments, units, cardinality, and PII-safe telemetry conventions | Signal selection intent, pipeline behavior, or drift interpretation | +| Exact skill name | Capability description | Does not own | +|-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------| +| `data-catalog` | Durable catalog entities, declared relationships, identity, coverage, lineage, and attached dataset profiles | Pipeline tier behavior, inferred diagram relationships, or feasibility verdicts | +| `dataops` | DataOps tier semantics, pipeline invariants, validation, tests, drift, signals, and derived-dataset persistence | Catalog meaning, experiment selection, ML tracking setup, or sensitivity classification | +| `feasibility` | Evidence-led feasibility studies, lifecycle, recommendations, and interchange traceability | Functional requirement numbering, downstream writeback, catalog semantics, or implementation | +| `analysis-authoring` | EDA notebook and analytical dashboard composition, visualization selection, and dashboard validation | Column semantics, persistence format, accessibility conformance, or evaluation dataset design | +| `evaluation-design` | AI-system evaluation dataset design, difficulty balance, metric selection, and tooling fit | Trained-model evaluation, Responsible AI approval, or session and durable-write mechanics | +| `experiment-design` | Problem-class framing for a stated business outcome, plus experiment selection, hypotheses, vetting, minimum scope, and result evaluation | ML infrastructure, production implementation, or pipeline mechanics | +| `ml-experimentation` | ML environments, reproducibility, tracking, abstractions, evaluation, and readiness | General hypothesis framing, pipeline replay, or Responsible AI approval | +| `privacy-standards` | Sensitivity classification, data-flow reasoning, standards mapping, and DPIA thresholds | Catalog semantics, pipeline mechanics, or durable-write authority | +| `rai-standards` | Responsible AI standards, risk framing, and standards mapping | Evaluation dataset design, model performance, or approval authority | +| `rai-planner` | Scoped Responsible AI assessment when a surfaced risk warrants one | Data science method authority, session mechanics, or approval authority | +| `telemetry-foundations` | Metric names, instruments, units, cardinality, and PII-safe telemetry conventions | Signal selection intent, pipeline behavior, or drift interpretation | The two Responsible AI authorities are sequential, not alternatives. `rai-standards` provides risk framing and standards alignment and is evaluated @@ -78,6 +79,26 @@ first whenever an AI or ML system is in scope. `rai-planner` provides a scoped assessment of one surfaced risk and is reached only after that risk is identified. +`problem-framing` and `experiment` both route to `experiment-design` and stay +distinct. `problem-framing` asks what kind of data-science problem a stated +business outcome is, and completes without choosing one. `experiment` assumes a +direction and selects what to test. + +## Problem-framing completion evidence + +This job has two completion shapes and neither is an abandonment. + +* Normal completion records the candidate problem classes, the grounded-versus- + original reasoning behind each, unresolved discriminating questions marked + unknown, and gaps carrying an answer holder and a coordinator. No candidate is + selected, ranked, or scored. +* An unstated business outcome completes with a single outcome-definition gap, + its answer holder, its coordinator, and an explicit statement that no candidate + analysis was performed. Record it as completed work, not `discarded-cleanly`. + +Completion returns control to the coach. A feasibility or experiment job starts +only after a separate user-confirmed transition. + The two evaluation authorities are adjacent and must not be blended. `evaluation-design` covers systems whose output is a response, such as assistants and agents. `ml-experimentation` covers a trained model's predictive diff --git a/.github/skills/data-science-engineering/data-science-engineering-foundation/references/session-state.md b/.github/skills/data-science-engineering/data-science-engineering-foundation/references/session-state.md index 285481095e..ecddf61522 100644 --- a/.github/skills/data-science-engineering/data-science-engineering-foundation/references/session-state.md +++ b/.github/skills/data-science-engineering/data-science-engineering-foundation/references/session-state.md @@ -51,6 +51,10 @@ jobs: class: episodic status: never invocations: [] + problem-framing: + class: episodic + status: never + invocations: [] feasibility: class: bounded status: never diff --git a/.github/skills/hve-core/c4-architecture/SKILL.md b/.github/skills/hve-core/c4-architecture/SKILL.md index 4738013ab2..59d70890cd 100644 --- a/.github/skills/hve-core/c4-architecture/SKILL.md +++ b/.github/skills/hve-core/c4-architecture/SKILL.md @@ -1,7 +1,7 @@ --- name: c4-architecture description: "Model and document existing or planned software architectures with the C4 model across System Context, Container, and Component levels plus deployment diagrams, then emit diagrams through a selected renderer. Use when an architect needs audience-appropriate software architecture documentation; use the 'architecture-diagrams' skill for infrastructure topology." -license: mixed +license: CC-BY-4.0 AND MIT user-invocable: true compatibility: "Generates Mermaid source in any chat context. Preview needs a Mermaid-capable Markdown host. CLI render validation needs Node.js, @mermaid-js/mermaid-cli (mmdc) on PATH plus a Chrome/Chromium runtime; install may need package-manager and network access." metadata: @@ -72,3 +72,11 @@ Each source is paraphrased with attribution; none is reproduced verbatim. - C4 model concepts and abstractions (System Context, Container, Component, Code) are based on the [C4 Model](https://c4model.com/) by Simon Brown, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). - Mermaid syntax conventions in the [Mermaid renderer](references/renderer-mermaid.md) are derived from the [Mermaid documentation](https://mermaid.js.org/), licensed under [MIT](https://github.com/mermaid-js/mermaid/blob/develop/LICENSE). + +Derived and original material share files here, so the boundary is marked inline in each file rather than split by file. The frontmatter expression is the conjunction of every license present in the package. + +| Path | License | Origin | +|-------------------------------------------|-------------------|------------------------------------------------------------------------------------------------| +| `references/renderer-mermaid.md` | CC-BY-4.0 AND MIT | Repository renderer rules, paraphrased C4 concepts, and MIT-derived Mermaid syntax conventions | +| `references/c4-modelling-instructions.md` | CC-BY-4.0 | Repository modelling rules with paraphrased C4 concepts | +| `SKILL.md` and remaining package content | CC-BY-4.0 | Repository-original with paraphrased C4 concepts | diff --git a/.github/skills/project-planning/experiment-design/SKILL.md b/.github/skills/project-planning/experiment-design/SKILL.md index c93ba1acdb..0b08434d54 100644 --- a/.github/skills/project-planning/experiment-design/SKILL.md +++ b/.github/skills/project-planning/experiment-design/SKILL.md @@ -1,13 +1,13 @@ --- name: experiment-design -description: "Experiment design reference for Minimum Viable Experiment coaching, hypothesis formation, vetting and red flags, and experiment readiness. Use when framing, vetting, scoping, or evaluating an experiment of any kind, including data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments." +description: "Experiment design reference for problem-class framing, Minimum Viable Experiment coaching, hypothesis formation, vetting and red flags, and experiment readiness. Use when translating a stated business outcome into candidate data-science problem classes, or when framing, vetting, scoping, or evaluating an experiment of any kind, including data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments." license: CC-BY-4.0 user-invocable: false metadata: - authors: "Microsoft (MVE coaching synthesis); Microsoft (Code With Engineering Playbook)" - spec_version: "1.0" - last_updated: "2026-08-03" - content_based_on: "HVE Core original MVE coaching material; https://microsoft.github.io/code-with-engineering-playbook/design/design-reviews/recipes/engineering-feasibility-spikes/; https://microsoft.github.io/code-with-engineering-playbook/design/design-reviews/trade-studies/" + authors: "Microsoft (MVE coaching synthesis); Microsoft (Code With Engineering Playbook); Google LLC (machine-learning problem framing)" + spec_version: "1.1" + last_updated: "2026-08-21" + content_based_on: "HVE Core original MVE coaching material; HVE Core original cross-paradigm routing; https://microsoft.github.io/code-with-engineering-playbook/design/design-reviews/recipes/engineering-feasibility-spikes/; https://microsoft.github.io/code-with-engineering-playbook/design/design-reviews/trade-studies/; https://developers.google.com/machine-learning/problem-framing/problem; https://developers.google.com/machine-learning/guides/rules-of-ml" --- # Experiment Design Reference Pack @@ -16,11 +16,16 @@ metadata: Support experiment work end to end: turning unknowns into testable hypotheses, screening out work that is not a real experiment, and scoping it so the result is comparable and decision-ready. +Support the step that precedes it as well: translating a stated business outcome into candidate data-science problem classes with the reasoning that produced them, so a practitioner knows what kind of problem is on the table before deciding what to test. + +The two concerns stay distinct. Problem-class framing exposes candidates and never selects one. Experiment work assumes a candidate direction already exists and concludes by selecting an experiment with the team. + This pack is general purpose. It applies to data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments, not to data science alone. ## Inputs * The problem statement, customer context, and business driver +* The stated business outcome, when the active concern is problem-class framing * Known unknowns, assumptions, and risks * The decision the experiment is meant to unblock * Prior experiment results, when a sequence of experiments is in flight @@ -31,12 +36,18 @@ Read only the reference that matches the active concern. | Reference | Read this when | |---------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [problem-framing.md](references/problem-framing.md) | Translating a stated business outcome into candidate data-science problem classes, applying per-paradigm entry tests, ordering discriminating questions, or recording assignable gaps | | [mve-coaching.md](references/mve-coaching.md) | Framing an MVE, forming or sharpening hypotheses, applying vetting criteria and red flags, designing the experiment, evaluating results, or producing session and backlog-bridge artifacts | | [experiment-readiness.md](references/experiment-readiness.md) | Deciding which experiment to run at all: turning a risk landscape into candidates, prioritizing among competing unknowns, comparing options with evidence, or re-prioritizing mid-flight | | [provenance.md](references/provenance.md) | Confirming what is upstream guidance, what is HVE Core derivation or repository convention, and where upstream is silent | +A confirmed problem-class framing request reads [problem-framing.md](references/problem-framing.md) only. Experiment requests read [mve-coaching.md](references/mve-coaching.md) or [experiment-readiness.md](references/experiment-readiness.md) by concern. + ## Success criteria +* Problem-class framing returns the surviving candidate problem classes with the entry-test reasoning that kept or discarded each one, and never returns a selection, a ranking, or a confidence score. +* Every framing claim is visibly either a cited entry test or repository-original routing, and no claim implies that a universal external standard governs cross-paradigm routing. +* An unanswered framing question becomes a recorded gap with an answer holder and a coordinator, and the remaining analysis continues. * Each hypothesis is testable, specific, falsifiable, and tied to a stated rationale. * The experiment selected is the one whose result changes the most downstream work, chosen from a candidate list rather than from the first idea proposed. * Work that is a demo, a mini-MVP, or an already-answered question is named as such rather than run as an experiment. @@ -48,20 +59,25 @@ Read only the reference that matches the active concern. * Reproduce only the minimum text necessary for a specific technical point, and paraphrase everything else. Attribute every reference and describe accurately what each reference reproduces. * Label repository conventions as substitutions rather than upstream recommendations. * Keep experiment framing separate from experiment infrastructure. This pack decides whether and how an experiment is worth running; it does not stand up tooling. +* Keep problem-class framing separate from experiment framing. Problem-class framing asks what kind of problem this is; experiment framing asks what to test about a direction already chosen. Do not apply experiment vetting criteria, red flags, or readiness prioritization to a confirmed problem-class framing request. +* Never rank, score, threshold, or recommend a candidate problem class. The practitioner is the subject-matter expert who chooses. ## Ownership boundaries -| Concern | Owner | -|-------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| -| MVE session directory, artifact filenames, placement, and tracking-file hygiene | `experiment-designer.instructions.md`, applied automatically to MVE tracking paths | -| Phase order, gates, session writes, and coaching flow | The consuming experiment agent | -| ML environments, reproducibility, tracking, model evaluation, abstractions, and readiness | `ml-experimentation`, the ML-specific experimentation reference | -| Pipeline mechanics, data tiering, replay, validation, and DS/MLOps test technique | `dataops`, the DataOps and testing reference | -| Metric names, instruments, units, cardinality, and PII-safe telemetry | `telemetry-foundations`, the OpenTelemetry-aligned instrumentation skill | -| Data sensitivity classification and DPIA thresholds | `privacy-standards`, the privacy classification reference | +| Concern | Owner | +|--------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| +| Candidate problem classes for a stated business outcome, entry-test reasoning, discriminating questions, and assignable framing gaps | This pack, through [problem-framing.md](references/problem-framing.md) | +| Whether a proposed outcome is achievable with available data and evidence | `feasibility`, the evidence-led feasibility study reference | +| MVE session directory, artifact filenames, placement, and tracking-file hygiene | `experiment-designer.instructions.md`, applied automatically to MVE tracking paths | +| Phase order, gates, session writes, and coaching flow | The consuming experiment agent | +| ML environments, reproducibility, tracking, model evaluation, abstractions, and readiness | `ml-experimentation`, the ML-specific experimentation reference | +| Pipeline mechanics, data tiering, replay, validation, and DS/MLOps test technique | `dataops`, the DataOps and testing reference | +| Metric names, instruments, units, cardinality, and PII-safe telemetry | `telemetry-foundations`, the OpenTelemetry-aligned instrumentation skill | +| Data sensitivity classification and DPIA thresholds | `privacy-standards`, the privacy classification reference | ## Stop rules +* For a confirmed problem-class framing request, an unstated business outcome is the sole stop before candidate analysis. [problem-framing.md](references/problem-framing.md) owns that stop and its job-specific exceptions. The experiment stop rules below govern experiment requests only. * Stop and name the red flag when the request is a demo, a scaled-down product build, or a question already answered elsewhere. Then offer either the falsifiable hypothesis hiding underneath it or an explicit non-experiment path, rather than halting on the refusal. * Stop and separate concerns when the request is production implementation rather than experiment design. * Stop and state the gap when the request depends on guidance this pack does not provide, such as a universal framework, tool, or metric choice made without project context. @@ -72,6 +88,8 @@ This pack declares `CC-BY-4.0`. [mve-coaching.md](references/mve-coaching.md) is repository-original content under CC BY 4.0. It is not derived from any upstream source and cites no upstream URL. +[problem-framing.md](references/problem-framing.md) layers three source treatments. Its machine-learning entry tests are informed by Google for Developers machine-learning guidance, licensed CC BY 4.0; that material is paraphrased and reorganized, both upstream URLs are cited, and changes are stated. It points readers to the NEOS Guide for optimization problem types as a citation only, and states no NEOS classification of its own. Its cross-paradigm routing, the broader analytical-fit judgement, and the gap and output contract are repository-original. + [experiment-readiness.md](references/experiment-readiness.md) is HVE Core guidance informed by two Microsoft CSE Code With Engineering Playbook documentation pages, which are licensed CC BY 4.0. It paraphrases rather than reproduces, generalizes the upstream practices beyond engagement-shaped engineering work, cites both upstream URLs, and states that changes were made. See [provenance.md](references/provenance.md) for the consolidated source map and derivation labels. diff --git a/.github/skills/project-planning/experiment-design/references/problem-framing.md b/.github/skills/project-planning/experiment-design/references/problem-framing.md new file mode 100644 index 0000000000..46ad5ec6fa --- /dev/null +++ b/.github/skills/project-planning/experiment-design/references/problem-framing.md @@ -0,0 +1,107 @@ +--- +title: Problem-class framing +description: Translating a stated business outcome into candidate data-science problem classes using cited entry tests, discriminating questions, and assignable gaps, without selecting a framing for the practitioner +--- + +## Sources + +* Google for Developers, [Introduction to Machine Learning Problem Framing: Understand the problem](https://developers.google.com/machine-learning/problem-framing/problem), licensed CC BY 4.0. +* Google for Developers, [Rules of Machine Learning](https://developers.google.com/machine-learning/guides/rules-of-ml), licensed CC BY 4.0. + +The machine-learning entry tests below are informed by those two pages and have been changed: they are paraphrased, reorganized into per-paradigm entry tests, restated in this repository's vocabulary, and placed inside a cross-paradigm procedure the upstream pages do not describe. `THIRD-PARTY-NOTICES` carries the attribution CC BY 4.0 requires. + +The NEOS Guide is an authority a reader can consult for optimization problem types, including [Optimization Problem Types](https://neos-guide.org/guide/types/), [Linear Programming](https://neos-guide.org/guide/types/lp/), [Integer Programming](https://neos-guide.org/guide/types/integer/), and [Nonlinear Constrained Optimization](https://neos-guide.org/guide/types/nonlin/). Its terms reserve rights, so it is cited only. No NEOS text, taxonomy, classification, diagram, or derivative of them is reproduced or paraphrased here. + +No external standard adjudicates across paradigms. Every cross-paradigm routing judgement, the broader analytical-fit disposition, the gap contract, and the output contract below are repository-original. + +## Read this when + +A practitioner has a business outcome and does not yet know what kind of data-science problem it is. This file produces the candidate problem classes and the reasoning behind them. It does not decide which one to pursue, estimate whether it is achievable, or design an experiment. + +## Provenance labels + +Carry these labels into the output so the practitioner can see what is grounded and what is judgement. + +| Label | Meaning | +|-------|-----------------------------------------------------------------------| +| `[G]` | Grounded in a cited source named in this file | +| `[O]` | Repository-original routing judgement; no external source adjudicates | +| `[A]` | Assumed from the conversation and not verified against data | + +## The one stop before analysis + +A stated business outcome is required before candidate analysis begins. It is the only stop this procedure applies before producing candidates. + +1. Coach for it first. Lead with the decision or change the framing should enable, then follow up on who acts on it and what would be different if it worked. +2. If the outcome is stated, proceed to the procedure. +3. If it stays unstated, do not analyze candidates and do not treat defining it as an experiment. Complete the invocation with a single outcome-definition gap carrying an answer holder and a coordinator, and record explicitly that no candidate analysis was performed. + +Experiment vetting criteria, experiment red flags, the unclear-problem-as-experiment fallback in [mve-coaching.md](mve-coaching.md), and readiness prioritization in [experiment-readiness.md](experiment-readiness.md) govern experiment requests. None of them applies to a confirmed problem-class framing request. + +## Procedure + +1. Restate the outcome for framing purposes and mark the restatement `[O]`. Name any ambiguity the restatement exposes; that ambiguity is usually the highest-power question in step 3. +2. Apply the entry test for every candidate problem class. Keep a class when its entry test passes or cannot yet be evaluated. Discard a class only when its entry test plainly fails, and record the failure reason rather than dropping it silently. +3. Order the unresolved questions by discriminating power: how many candidates the answer eliminates, not how easy it is to answer. +4. Work the questions in that order. Ask the practitioner for each value before recording it as unknown. +5. When elicitation does not produce a value, mark it unknown, record the gap, and continue with the remaining questions. An unanswered question never terminates the run. +6. Produce the output contract. Present the surviving candidates, the entry-test reasoning for each, the discriminating questions with their current state, the gap list, and any terminal disposition. + +## Candidate problem classes and entry tests + +| Candidate class | Entry test | Label | +|---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|-------| +| Supervised classification | A categorical outcome is labelled for past cases, and the features exist at the moment the prediction must be made | `[G]` | +| Supervised regression | The same conditions hold with a continuous target rather than a categorical one | `[G]` | +| Time-to-event analysis | The outcome is when something happens rather than whether it happened, and some observations are still in progress rather than negative | `[O]` | +| Forecasting | The target is time ordered, past behavior is a plausible basis for extrapolation, and the forecast itself does not materially change the outcome | `[O]` | +| Causal inference | The question is what happens if an input is set to a value, not what has historically co-occurred with that value | `[O]` | +| Constrained optimization | The task is choosing decision variables against a stated objective under stated constraints, rather than estimating an unknown quantity | `[O]` | +| Reinforcement learning | An agent acts repeatedly in an environment, observes a reward, and has enough interaction volume for a policy to be learned | `[O]` | +| Generative modelling | The required output is new content conditioned on context, and quality is judged against an evaluation rubric rather than a labelled target | `[O]` | +| Not a modelling problem | The population is too small to support inference, or the value lies in shared understanding rather than a fitted model | `[O]` | + +### Machine learning versus a non-machine-learning approach + +Before committing to any modelling class, compare the machine-learning approach against a non-machine-learning baseline on the value it adds, its cost and ongoing maintenance, and the resources it requires. A predictive machine-learning framing additionally requires data that carries usable predictive signal and a prediction someone can act on. Starting with a heuristic and moving to machine learning only once that heuristic becomes hard to maintain is a defensible path `[G]`. + +This test decides machine learning versus not machine learning. It does not decide whether the problem is a data problem at all. Statistics, optimization, and decision analysis are outside its scope, and no cited source adjudicates them; the broader analytical-fit judgement is `[O]` and must be presented as such. + +### Optimization structure + +When a constrained-optimization framing survives, its mathematical structure narrows the work further. This procedure does not classify that structure. Record the structural question as a discriminating question with its own gap `[O]`, and point the practitioner to the NEOS Guide pages cited above. + +## Gaps + +Every unanswered question becomes a gap. A gap is a coordinator handoff, not a practitioner to-do list. + +| Field | Content | +|----------------|-----------------------------------------------------------------------------| +| Needed fact | The specific value or definition required | +| Why it matters | Which candidates the answer keeps or eliminates | +| Answer holder | The person or role who holds the fact, or `unassigned` when none is known | +| Coordinator | The person who will obtain it, often not the answer holder, or `unassigned` | +| Status | `unknown` until a value is supplied | + +Record `unassigned` rather than a guess whenever an answer holder or a coordinator is not known, and name the unassigned ownership in the completion evidence. An unowned gap is a finding, not an omission, and it does not prevent the invocation from completing. + +## Output contract + +A completed framing invocation returns: + +* The restated outcome, labelled `[O]`. +* The surviving candidate classes, each with its entry test, its provenance label, and its current status. +* Discarded classes with the entry-test failure that removed them. +* The discriminating questions in power order, with current answers or `unknown`. +* The gap table. +* Any terminal disposition, such as a non-machine-learning `[G]` or a not-a-modelling-problem `[O]` result. + +An invocation stopped for an unstated business outcome returns the outcome-definition gap and an explicit statement that no candidate analysis was performed. + +## What this procedure never does + +* It never selects, ranks, scores, or confidence-thresholds a candidate. The practitioner is the subject-matter expert and makes that call. +* It never estimates feasibility. `feasibility` owns whether a proposed outcome is achievable with the available data and evidence. +* It never designs or scopes an experiment. That begins after a framing direction is chosen. +* It never invents a missing fact. Unknowns are labelled and assigned. +* It never presents an original routing judgement as though a standard supports it. diff --git a/.github/skills/project-planning/experiment-design/references/provenance.md b/.github/skills/project-planning/experiment-design/references/provenance.md index 29ebe5425e..adee6e7147 100644 --- a/.github/skills/project-planning/experiment-design/references/provenance.md +++ b/.github/skills/project-planning/experiment-design/references/provenance.md @@ -15,11 +15,17 @@ The pack previously carried two Microsoft CSE Code With Engineering Playbook ref Microsoft CSE Code With Engineering Playbook documentation is licensed CC BY 4.0, which requires attribution and an indication that changes were made; `THIRD-PARTY-NOTICES` carries both. This repository's licensing posture is stricter than the license and limits reproduction to the minimum text necessary for a specific technical point. [experiment-readiness.md](experiment-readiness.md) satisfies that limit: it paraphrases throughout and reproduces no upstream passage. +Google for Developers machine-learning documentation is licensed CC BY 4.0 under the same requirements, and `THIRD-PARTY-NOTICES` carries its attribution and change statement. The NEOS Guide reserves rights and is therefore cited only: no NEOS text, taxonomy, diagram, or derivative of them appears in this pack, and no content claims derivation from it. + ## Source map | Content area | Source | Reproduction scope | |-------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------| | MVE methodology, hypothesis format, vetting criteria, red flags, experiment types, design practices, results evaluation, and the backlog bridge | HVE Core original material | Not derived from any upstream source | +| Machine-learning versus non-machine-learning comparison, and the supervised entry conditions of labelled outcomes and prediction-time features | [Understand the problem](https://developers.google.com/machine-learning/problem-framing/problem) | Paraphrased and reorganized into per-paradigm entry tests; no passage reproduced | +| Beginning with a heuristic and moving to machine learning once the heuristic becomes hard to maintain | [Rules of Machine Learning](https://developers.google.com/machine-learning/guides/rules-of-ml) | Paraphrased; no passage reproduced | +| Optimization problem types, as further reading only | [NEOS Guide](https://neos-guide.org/guide/types/) | Cited only; nothing reproduced, paraphrased, classified, or derived | +| Cross-paradigm routing, the broader analytical-fit judgement, and the gap and output contracts | HVE Core original material | Not derived from any upstream source | | Candidate generation from a failure exercise, and the mid-flight share that re-prioritizes the next experiment | [Engineering Feasibility Spikes](https://microsoft.github.io/code-with-engineering-playbook/design/design-reviews/recipes/engineering-feasibility-spikes/) | Paraphrased and generalized; upstream terms named for findability; no passage reproduced | | Evidence-based comparison of competing options, and when not to run one | [Trade Studies](https://microsoft.github.io/code-with-engineering-playbook/design/design-reviews/trade-studies/) | Paraphrased and generalized; upstream terms named for findability; no passage reproduced | @@ -32,14 +38,19 @@ MVE coaching content in this pack is HVE Core original material. It was consolid ## HVE Core derivations -| Item | Position | -|------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------| -| MVE methodology, vetting criteria, red flags, and backlog bridge | HVE Core original coaching material, consolidated into this pack | -| Prioritization signals for choosing among candidate experiments | HVE Core guidance. Upstream describes re-prioritizing after each share but does not enumerate selection signals. | -| Generalizing spikes and trade studies beyond engagement work | HVE Core framing. Upstream scopes both practices to customer engagements with named ceremonies and meeting lengths. | +| Item | Position | +|-------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------| +| MVE methodology, vetting criteria, red flags, and backlog bridge | HVE Core original coaching material, consolidated into this pack | +| Cross-paradigm routing among candidate problem classes | HVE Core original. No external standard adjudicates across paradigms; the cited source is machine-learning scoped. | +| The broader not-a-modelling-problem judgement | HVE Core original. The cited source decides machine learning versus not machine learning only. | +| Optimization problem-type classification | Not stated by this pack. The framing procedure records the structural question and cites NEOS as further reading. | +| The gap contract with an answer holder and a separate coordinator | HVE Core original, established by prototype testing rather than upstream guidance | +| Prioritization signals for choosing among candidate experiments | HVE Core guidance. Upstream describes re-prioritizing after each share but does not enumerate selection signals. | +| Generalizing spikes and trade studies beyond engagement work | HVE Core framing. Upstream scopes both practices to customer engagements with named ceremonies and meeting lengths. | ## Where this pack is silent +* Whether a proposed outcome is achievable with the available data and evidence belongs to `feasibility`. This pack exposes candidate problem classes; it does not assess them. * ML experimentation setup, experiment tracking frameworks, dataset and model abstractions, model evaluation flow, and ML checklists belong to `ml-experimentation`. * Pipeline mechanics, data tiering, replay semantics, and DS/MLOps test technique belong to `dataops`. * This pack does not select a universal framework, tool, metric, or threshold. It requires that a team decide, document, and apply one consistently, and that decision depends on project context. diff --git a/.github/skills/project-planning/functional-planner/SKILL.md b/.github/skills/project-planning/functional-planner/SKILL.md index 8ae66032ee..225202a54a 100644 --- a/.github/skills/project-planning/functional-planner/SKILL.md +++ b/.github/skills/project-planning/functional-planner/SKILL.md @@ -1,7 +1,7 @@ --- name: functional-planner description: "Read-only PRD-to-work-item hierarchy planning. Use to turn a PRD into a validated Azure DevOps, GitHub, or Jira handoff." -license: mixed +license: CC-BY-4.0 AND CC-BY-SA-4.0 user-invocable: true argument-hint: "[prd path or description] [platform=ado|github|jira] [lens=generic|scrum|kanban]" compatibility: "Hosts: vscode, github-coding-agent. Requires read access to the target tracker (Azure DevOps, GitHub, or Jira); for Jira, JIRA_BASE_URL plus JIRA_API_TOKEN or JIRA_PAT." @@ -144,3 +144,17 @@ The plan hands off to the `Backlog Manager` for a separate execution pass: ## Untrusted Content Boundary The Untrusted Content Boundary in the `backlog-management` skill governs item bodies, comments, and fetched platform payloads. This skill adds one subject: PRD text is untrusted content too, so a requirement written into a PRD never redirects the workflow, widens its scope, or triggers a mutation. + +## Attribution and licensing + +The two framework lenses paraphrase third-party guides with attribution; neither reproduces its source. Both sources are ShareAlike, which propagates to the paraphrase. Every other file is repository-original. The frontmatter expression is the conjunction of every license present in the package. + +| Path | License | Origin | +|------------------------------------------|--------------|--------------------------------------------------------------------| +| `references/frameworks/scrum.md` | CC-BY-SA-4.0 | Paraphrase of The Scrum Guide (2020), CC BY-SA 4.0 | +| `references/frameworks/kanban.md` | CC-BY-SA-4.0 | Paraphrase of The Kanban Guide (May 2025), treated as CC BY-SA 4.0 | +| `references/frameworks/generic.md` | CC-BY-4.0 | Repository-original | +| `references/ado.md` | CC-BY-4.0 | Repository-original | +| `references/github.md` | CC-BY-4.0 | Repository-original | +| `references/jira.md` | CC-BY-4.0 | Repository-original | +| `SKILL.md` and remaining package content | CC-BY-4.0 | Repository-original | diff --git a/.github/skills/project-planning/privacy-standards/SKILL.md b/.github/skills/project-planning/privacy-standards/SKILL.md index f83ac2286e..9fc16b8937 100644 --- a/.github/skills/project-planning/privacy-standards/SKILL.md +++ b/.github/skills/project-planning/privacy-standards/SKILL.md @@ -1,7 +1,7 @@ --- name: privacy-standards description: "Privacy planning reference for data-flow reasoning, standards mapping, and DPIA thresholds" -license: mixed +license: CC-BY-4.0 AND CC-BY-SA-3.0 user-invocable: false metadata: authors: "NIST (Privacy Framework and NISTIR 8062); GDPR and CCPA/CPRA sources; OWASP (privacy risks); Microsoft (planning synthesis)" @@ -19,9 +19,22 @@ This skill is the reusable privacy reference package for the Privacy Planner and ## Attribution and licensing posture -- NIST Privacy Framework and NISTIR 8062 are U.S. Government documents and are referenced here with attribution as public-domain reference material. -- GDPR and CCPA/CPRA content is paraphrased and attributed rather than quoted verbatim, consistent with the repository's open legal-text posture. -- OWASP privacy-risk material is used as a planning reference and is attributed to the OWASP project. +- NIST Privacy Framework and NISTIR 8062 are U.S. Government documents and are referenced here with attribution as public-domain reference material. Public-domain material carries no license obligation, so it adds no term to the package license expression. +- GDPR and CCPA/CPRA content is paraphrased and attributed rather than quoted verbatim, consistent with the repository's open legal-text posture. The paraphrase is this repository's own expression. +- OWASP privacy-risk material is adapted from the OWASP Top 10 Privacy Risks project, which is licensed CC BY-SA 3.0. ShareAlike propagates to that derivative, so it is the one file carrying a copyleft term. + +The frontmatter expression is the conjunction of every license present in the package. The table states which license governs each file. + +| Path | License | Origin | +|--------------------------------------------|--------------|-------------------------------------------------------------| +| `references/owasp-top-10-privacy-risks.md` | CC-BY-SA-3.0 | Adapted from OWASP Top 10 Privacy Risks v2.0 (CC BY-SA 3.0) | +| `references/nist-privacy-framework.md` | CC-BY-4.0 | Adapted from NIST Privacy Framework v1.0 (public domain) | +| `references/nistir-8062.md` | CC-BY-4.0 | Adapted from NISTIR 8062 (public domain) | +| `references/gdpr.md` | CC-BY-4.0 | Paraphrase of GDPR, not reproduced | +| `references/ccpa-cpra.md` | CC-BY-4.0 | Paraphrase of CCPA/CPRA, not reproduced | +| `references/dpia-thresholds.md` | CC-BY-4.0 | Repository-original planning synthesis | +| `references/data-governance-controls.md` | CC-BY-4.0 | Repository-original planning synthesis | +| `SKILL.md` and remaining package content | CC-BY-4.0 | Repository-original | ## Framework index diff --git a/.github/skills/rai/rai-standards/SKILL.md b/.github/skills/rai/rai-standards/SKILL.md index cecba976e6..c9ed418274 100644 --- a/.github/skills/rai/rai-standards/SKILL.md +++ b/.github/skills/rai/rai-standards/SKILL.md @@ -1,7 +1,7 @@ --- name: rai-standards description: "Consolidated Responsible AI standards reference: NIST AI RMF 1.0, AI STRIDE threat-modeling overlay, EU AI Act risk tiers, and an open-standards catalog with phase mapping" -license: mixed +license: CC-BY-4.0 user-invocable: false metadata: authors: "NIST (AI RMF, public domain); EU (AI Act, paraphrased with attribution); Microsoft (AI STRIDE overlay)" @@ -16,10 +16,22 @@ This skill is the reusable standards package for the RAI Planner. It consolidate ## Attribution and licensing posture -- NIST AI RMF 1.0 is a U.S. Government document and is reproduced here as public-domain reference material with attribution. -- EU AI Act content in this skill is paraphrased and attributed rather than quoted verbatim, consistent with the open legal-text posture used in the repository. +- NIST AI RMF 1.0 is a U.S. Government document and is reproduced here as public-domain reference material with attribution. Public-domain material carries no license obligation, so it adds no term to the package license expression. +- EU AI Act content in this skill is paraphrased and attributed rather than quoted verbatim, consistent with the open legal-text posture used in the repository. The paraphrase is this repository's own expression. - The AI STRIDE overlay is Microsoft-authored reference material for threat-modeling reuse in the RAI workflow. +Every file in this package is therefore governed by the repository-original license. The table records the source each file draws on. + +| Path | License | Origin | +|------------------------------------------|-----------|---------------------------------------------------| +| `references/nist-ai-rmf-govern.md` | CC-BY-4.0 | Adapted from NIST AI RMF 1.0 (public domain) | +| `references/nist-ai-rmf-map.md` | CC-BY-4.0 | Adapted from NIST AI RMF 1.0 (public domain) | +| `references/nist-ai-rmf-measure.md` | CC-BY-4.0 | Adapted from NIST AI RMF 1.0 (public domain) | +| `references/nist-ai-rmf-manage.md` | CC-BY-4.0 | Adapted from NIST AI RMF 1.0 (public domain) | +| `references/eu-ai-act.md` | CC-BY-4.0 | Paraphrase of EU AI Act 2024/1689, not reproduced | +| `references/ai-stride-overlay.md` | CC-BY-4.0 | Repository-original | +| `SKILL.md` and remaining package content | CC-BY-4.0 | Repository-original | + ## Framework index - [NIST AI RMF - Govern](references/nist-ai-rmf-govern.md) diff --git a/.github/skills/security/vex/SKILL.md b/.github/skills/security/vex/SKILL.md index 89dac34b47..3e8cb701a5 100644 --- a/.github/skills/security/vex/SKILL.md +++ b/.github/skills/security/vex/SKILL.md @@ -88,9 +88,18 @@ Products use [Package URL (PURL)](https://github.com/package-url/purl-spec) form The OpenVEX specification reference content in this skill is derived from the OpenVEX Community specification and remains attributed to the OpenVEX Community. The reusable VEX management -playbooks and the surrounding guidance in this skill are hve-core-authored content. The skill -frontmatter uses a mixed-attribution metadata set so the upstream specification reference and the -hve-core playbooks are clearly distinguished. +playbooks and the surrounding guidance in this skill are hve-core-authored content. + +Licenses are allocated per file in the table below. The frontmatter expression is the conjunction +of every license present in the package, so a redistributor of the whole package complies with all +of them; the table states which license actually governs each file. + +| Path | License | Origin | +|------------------------------------------------------|------------|---------------------------| +| `references/openvex-schema.md` | Apache-2.0 | Derived from OpenVEX spec | +| `references/vex-status-logic.md` | Apache-2.0 | Derived from OpenVEX spec | +| `references/cve-data-sources.md` | CC-BY-4.0 | hve-core-authored | +| `SKILL.md`, playbooks, and remaining package content | CC-BY-4.0 | hve-core-authored | ### Third-Party Attribution diff --git a/THIRD-PARTY-NOTICES b/THIRD-PARTY-NOTICES index 0ae1ac156a..5dcd207479 100644 --- a/THIRD-PARTY-NOTICES +++ b/THIRD-PARTY-NOTICES @@ -68,6 +68,25 @@ retained as identifiers. --- +Google for Developers machine-learning documentation +Copyright: © Google LLC +License: Creative Commons Attribution 4.0 International (CC BY 4.0) +License URI: +Source: +Source: +Usage: Machine-learning problem-framing entry tests embedded in the experiment-design +skill's problem-framing reference: the comparison of a machine-learning approach against +a non-machine-learning baseline on value, cost and maintenance, and resources; the +predictive-machine-learning conditions of usable predictive data and an actionable +prediction; the supervised entry conditions of labelled outcomes and prediction-time +features; and preferring a heuristic until it becomes hard to maintain. The material has +been changed: it is paraphrased, reorganized into per-paradigm entry tests, restated in +this repository's vocabulary, and placed inside a cross-paradigm procedure the upstream +pages do not describe. No upstream passage is reproduced. Content available under the +Creative Commons Attribution 4.0 License except as otherwise noted by the source. + +--- + C4 Model Creator: Simon Brown License: Creative Commons Attribution 4.0 International (CC BY 4.0) diff --git a/docs/reference/skills/README.md b/docs/reference/skills/README.md index 5d35c014e1..a8520ffaf4 100644 --- a/docs/reference/skills/README.md +++ b/docs/reference/skills/README.md @@ -3,7 +3,7 @@ title: Skills description: Reference documentation for HVE Core skills. sidebar_position: 0 author: Microsoft -ms.date: 2026-08-20 +ms.date: 2026-08-28 ms.topic: overview keywords: - reference @@ -54,7 +54,7 @@ This page lists the generated reference documentation for HVE Core skills. | [backlog-execute](project-planning/backlog-execute.md) | Mutating backlog execution for Azure DevOps, GitHub, and Jira. Use to create one item or apply a reviewed handoff to a confirmed tracker. | | [backlog-management](project-planning/backlog-management.md) | Shared backlog conventions for Azure DevOps, GitHub, and Jira. Use for platform resolution, autonomy tiers, sanitization guards, and story quality. | | [backlog-plan](project-planning/backlog-plan.md) | Read-only backlog planning for Azure DevOps, GitHub, and Jira. Use to discover, triage, sprint-plan, or resume without mutating a tracker. | -| [experiment-design](project-planning/experiment-design.md) | Experiment design reference for Minimum Viable Experiment coaching, hypothesis formation, vetting and red flags, and experiment readiness. Use when framing, vetting, scoping, or evaluating an experiment of any kind, including data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments. | +| [experiment-design](project-planning/experiment-design.md) | Experiment design reference for problem-class framing, Minimum Viable Experiment coaching, hypothesis formation, vetting and red flags, and experiment readiness. Use when translating a stated business outcome into candidate data-science problem classes, or when framing, vetting, scoping, or evaluating an experiment of any kind, including data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments. | | [functional-planner](project-planning/functional-planner.md) | Read-only PRD-to-work-item hierarchy planning. Use to turn a PRD into a validated Azure DevOps, GitHub, or Jira handoff. | | [gitlab](project-planning/gitlab.md) | Manage GitLab merge requests and pipelines with a Python CLI | | [jira](project-planning/jira.md) | Jira issue workflows for search, issue updates, transitions, comments, field discovery, and interactive credential setup via the Jira REST API. Use when you need to configure Jira access, search with JQL, inspect an issue, create or update work items, move an issue between statuses, post comments, or discover required fields for issue creation. | diff --git a/docs/reference/skills/project-planning/experiment-design.md b/docs/reference/skills/project-planning/experiment-design.md index 2e533bd885..3489694bf9 100644 --- a/docs/reference/skills/project-planning/experiment-design.md +++ b/docs/reference/skills/project-planning/experiment-design.md @@ -1,9 +1,9 @@ --- title: experiment-design -description: "Experiment design reference for Minimum Viable Experiment coaching, hypothesis formation, vetting and red flags, and experiment readiness. Use when framing, vetting, scoping, or evaluating an experiment of any kind, including data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments." +description: "Experiment design reference for problem-class framing, Minimum Viable Experiment coaching, hypothesis formation, vetting and red flags, and experiment readiness. Use when translating a stated business outcome into candidate data-science problem classes, or when framing, vetting, scoping, or evaluating an experiment of any kind, including data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments." sidebar_position: 5 author: Microsoft -ms.date: 2026-08-19 +ms.date: 2026-08-28 ms.topic: reference keywords: - skill @@ -23,7 +23,7 @@ keywords: ## What it does -Experiment design reference for Minimum Viable Experiment coaching, hypothesis formation, vetting and red flags, and experiment readiness. Use when framing, vetting, scoping, or evaluating an experiment of any kind, including data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments. +Experiment design reference for problem-class framing, Minimum Viable Experiment coaching, hypothesis formation, vetting and red flags, and experiment readiness. Use when translating a stated business outcome into candidate data-science problem classes, or when framing, vetting, scoping, or evaluating an experiment of any kind, including data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments. ## When to use it diff --git a/evals/agent-behavior/eval.yaml b/evals/agent-behavior/eval.yaml index 08b0f296d7..66937a1715 100644 --- a/evals/agent-behavior/eval.yaml +++ b/evals/agent-behavior/eval.yaml @@ -841,6 +841,87 @@ stimuli: config: pattern: (?i)(planner\s+state\.json|\.copilot-tracking[/\\]ds[/\\][^\s]+[/\\]state\.json) negate: true +- name: data-science-engineering-coach-problem-framing-missing-outcome + prompt: | + I confirm the `problem-framing` job. The request is: "We want to do + something with our engagement data." No business outcome is stated, and I + decline to define one when you ask. + + State which skill owns the work, what you produce, and what happens next. + tags: + category: agent-behavior + advisory: "true" + agent: data-science-engineering-coach + graders: + - type: output-matches + name: framing-routes-to-experiment-design + config: + pattern: (?is)problem-framing.*experiment-design.*episodic + - type: output-matches + name: assigns-outcome-definition-gap + config: + pattern: (?is)(outcome[- ]definition|define the (business )?outcome).*?gap + - type: output-matches + name: gap-carries-answer-holder-and-coordinator + config: + pattern: (?is)answer holder.*?coordinator + - type: output-matches + name: performs-no-candidate-analysis + config: + pattern: (?is)(no candidate analysis|no analysis was performed|without analysis|does not analyze|before candidate analysis) + - type: output-matches + name: completes-rather-than-discarding + config: + pattern: (?is)(completed work|completes|completion|complete the invocation) + - type: output-matches + name: missing-outcome-does-not-become-an-experiment + config: + pattern: (?is)(does not|will not|rather than|instead of|no)\s+\w*\s*(start|become|becomes|treat|design|run|scope)[^.]{0,60}(MVE|minimum viable experiment|experiment) + - type: output-matches + name: user-confirmed-next-job + config: + pattern: (?is)(ask|offer|confirm|choose|would you like).*?(next|another).*?(job|feasibility|experiment) +- name: data-science-engineering-coach-problem-framing-unresolved-axis + prompt: | + I confirm the `problem-framing` job. The stated outcome is: "Determine + whether pre-engagement inputs predict whether an engagement succeeds." + Nobody can tell you whether the intent is to anticipate outcomes or to set + those inputs as policy. + + State which skill owns the work, what you produce, and what happens next. + tags: + category: agent-behavior + advisory: "true" + agent: data-science-engineering-coach + graders: + - type: output-matches + name: framing-routes-to-experiment-design + config: + pattern: (?is)problem-framing.*experiment-design.*episodic + - type: output-matches + name: marks-unknown-and-continues + config: + pattern: (?is)unknown.*?(continue|proceed|remaining) + - type: output-matches + name: gap-carries-answer-holder-and-coordinator + config: + pattern: (?is)answer holder.*?coordinator + - type: output-matches + name: candidates-carry-grounded-and-original-labels + config: + pattern: (?is)(\[G\]|grounded|cited).*?(\[O\]|repository-original|original routing|original judgement) + - type: output-matches + name: no-candidate-selected + config: + pattern: (?is)(does not select|do not select|not select|without selecting|no candidate is selected|selects none|no recommendation|does not rank|not rank|no ranking|not ranked) + - type: output-matches + name: completes-episodically + config: + pattern: (?is)(episodic|completes|completion|completed).*?(invocation|job|work) + - type: output-matches + name: user-confirmed-next-job + config: + pattern: (?is)(ask|offer|confirm|choose|would you like).*?(next|another).*?(job|feasibility|experiment) - name: data-science-engineering-coach-durable-write-scan prompt: | I selected catalog enrichment and approved a customer-repository output diff --git a/evals/agent-behavior/expectations/data-science-engineering-coach.expectations.yml b/evals/agent-behavior/expectations/data-science-engineering-coach.expectations.yml index 106d1efd9a..fe31983639 100644 --- a/evals/agent-behavior/expectations/data-science-engineering-coach.expectations.yml +++ b/evals/agent-behavior/expectations/data-science-engineering-coach.expectations.yml @@ -4,7 +4,7 @@ slug: data-science-engineering-coach class: planner-coach agent_file: .github/agents/data-science-engineering/data-science-engineering-coach.agent.md stimulus_file: evals/agent-behavior/stimuli/data-science-engineering-coach.yml -source_review_date: 2026-08-03 +source_review_date: 2026-08-21 expectations: - expectation_id: foundation-loaded-explicitly @@ -162,6 +162,27 @@ expectations: priority: high contract_ref: "agent §Job routing; foundation references/job-registry.md §Registry" + - expectation_id: problem-framing-job-contract + summary: A confirmed problem-framing job exposes candidate problem classes without selecting one and stops only for an unstated business outcome. + signal: Output routes `problem-framing` to `experiment-design`, records assignable gaps, and offers no recommendation. + pass_criteria: | + A confirmed `problem-framing` job routes to `experiment-design` as an + episodic invocation. An unstated business outcome stops before candidate + analysis and completes with an outcome-definition gap carrying an answer + holder and a coordinator. A stated outcome with an unresolved high-power + question elicits the value, marks it unknown, records its answer holder + and coordinator, continues the remaining analysis, presents surviving + candidates with grounded-versus-original reasoning, selects none, + completes episodically, and asks before any next job. + failure_modes: + - Treats an undefined business outcome as the subject of an experiment. + - Recommends, ranks, or confidence-scores a candidate framing. + - Halts the analysis on an unanswered question. + - Applies experiment vetting, red flags, or readiness prioritization to a confirmed problem-framing job. + - Auto-advances into feasibility or experiment work. + priority: high + contract_ref: "foundation references/job-registry.md §Problem-framing completion evidence; experiment-design references/problem-framing.md" + - expectation_id: durable-write-scan-gate summary: Every durable customer-artifact write is scanned in data mode first. signal: Output describes scan-before-write, blocking high findings, and warning review. diff --git a/evals/agent-behavior/stimuli/data-science-engineering-coach.yml b/evals/agent-behavior/stimuli/data-science-engineering-coach.yml index afb690a7d9..29ba712158 100644 --- a/evals/agent-behavior/stimuli/data-science-engineering-coach.yml +++ b/evals/agent-behavior/stimuli/data-science-engineering-coach.yml @@ -111,6 +111,87 @@ stimuli: pattern: '(?i)(planner\s+state\.json|\.copilot-tracking[/\\]ds[/\\][^\s]+[/\\]state\.json)' negate: true + - name: data-science-engineering-coach-problem-framing-missing-outcome + prompt: | + I confirm the `problem-framing` job. The request is: "We want to do + something with our engagement data." No business outcome is stated, and I + decline to define one when you ask. + + State which skill owns the work, what you produce, and what happens next. + tags: + category: agent-behavior + advisory: "true" + graders: + - type: output-matches + name: framing-routes-to-experiment-design + config: + pattern: '(?is)problem-framing.*experiment-design.*episodic' + - type: output-matches + name: assigns-outcome-definition-gap + config: + pattern: '(?is)(outcome[- ]definition|define the (business )?outcome).*?gap' + - type: output-matches + name: gap-carries-answer-holder-and-coordinator + config: + pattern: '(?is)answer holder.*?coordinator' + - type: output-matches + name: performs-no-candidate-analysis + config: + pattern: '(?is)(no candidate analysis|no analysis was performed|without analysis|does not analyze|before candidate analysis)' + - type: output-matches + name: completes-rather-than-discarding + config: + pattern: '(?is)(completed work|completes|completion|complete the invocation)' + - type: output-matches + name: missing-outcome-does-not-become-an-experiment + config: + pattern: '(?is)(does not|will not|rather than|instead of|no)\s+\w*\s*(start|become|becomes|treat|design|run|scope)[^.]{0,60}(MVE|minimum viable experiment|experiment)' + - type: output-matches + name: user-confirmed-next-job + config: + pattern: '(?is)(ask|offer|confirm|choose|would you like).*?(next|another).*?(job|feasibility|experiment)' + + - name: data-science-engineering-coach-problem-framing-unresolved-axis + prompt: | + I confirm the `problem-framing` job. The stated outcome is: "Determine + whether pre-engagement inputs predict whether an engagement succeeds." + Nobody can tell you whether the intent is to anticipate outcomes or to set + those inputs as policy. + + State which skill owns the work, what you produce, and what happens next. + tags: + category: agent-behavior + advisory: "true" + graders: + - type: output-matches + name: framing-routes-to-experiment-design + config: + pattern: '(?is)problem-framing.*experiment-design.*episodic' + - type: output-matches + name: marks-unknown-and-continues + config: + pattern: '(?is)unknown.*?(continue|proceed|remaining)' + - type: output-matches + name: gap-carries-answer-holder-and-coordinator + config: + pattern: '(?is)answer holder.*?coordinator' + - type: output-matches + name: candidates-carry-grounded-and-original-labels + config: + pattern: '(?is)(\[G\]|grounded|cited).*?(\[O\]|repository-original|original routing|original judgement)' + - type: output-matches + name: no-candidate-selected + config: + pattern: '(?is)(does not select|do not select|not select|without selecting|no candidate is selected|selects none|no recommendation|does not rank|not rank|no ranking|not ranked)' + - type: output-matches + name: completes-episodically + config: + pattern: '(?is)(episodic|completes|completion|completed).*?(invocation|job|work)' + - type: output-matches + name: user-confirmed-next-job + config: + pattern: '(?is)(ask|offer|confirm|choose|would you like).*?(next|another).*?(job|feasibility|experiment)' + - name: data-science-engineering-coach-durable-write-scan prompt: | I selected catalog enrichment and approved a customer-repository output diff --git a/evals/behavior-conformance/skill-behavior.eval.yaml b/evals/behavior-conformance/skill-behavior.eval.yaml index 78f70d3265..e90854beb0 100644 --- a/evals/behavior-conformance/skill-behavior.eval.yaml +++ b/evals/behavior-conformance/skill-behavior.eval.yaml @@ -2371,6 +2371,25 @@ stimuli: config: pattern: '(?i)(vex|openvex|status|justification|not_affected|under_investigation)' + - name: skill-rai-standards-knowledge + prompt: | + Summarize what the `rai-standards` skill provides and name one Responsible + AI framework it references. + tags: + category: behavior-conformance + skill: rai-standards + shape: knowledge + advisory: "true" + graders: + - type: output-matches + name: skill-attribution + config: + pattern: '(?i)(rai|responsible ai|standards|reference)' + - type: output-matches + name: scope-language + config: + pattern: '(?i)(nist|ai rmf|stride|eu ai act|risk tier|govern|map|measure|manage)' + - name: skill-copilot-otel-metrics-knowledge prompt: | Summarize what the `copilot-otel-metrics` skill sets up and name the