Agent Skills: open-source-audit + security-audit, and the repo-audits store (ADR-009)#68
Conversation
…heck - Open-Sourcing-Software.md: replace hard-coded model names (Opus 4.8 / GPT-5.5) with capability-based 'current frontier model' wording, and clarify the audit is triggered by a repo owner requesting the GitHub Enterprise/org owner to make the repo public, as part of this process. - SKILL.md: link back to the open-sourcing process, note the operational trigger, and state the skill is model-agnostic. - Agent Skills/README.md: document how to write maximally portable skills and how to supply them to Claude, GPT and Gemini agents. - SKILL.md: add a documentation-presence check (missing = FAIL) tied to the 'Provide Documentation' principle. - SKILL.md: make 'European Union' the explicit copyright holder term.
There was a problem hiding this comment.
Pull request overview
Adds an “Agent Skills” section to Codex and introduces a new pre-publication-check skill to serve as the final technical audit gate before making an ECMWF repository public, and wires this skill into the existing open-sourcing process documentation.
Changes:
- Add
Agent Skills/pre-publication-checkskill describing a structured pre-publication technical audit and report format. - Link the Agent Skills section from the root
README.md. - Update open-sourcing guidance to require running the pre-publication check (with a current frontier model) before switching a repository to public.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| README.md | Adds a link to the new “Agent Skills” section from the main contents list. |
| Legal/Open-Sourcing-Software.md | Updates the open-sourcing checklist to require the pre-publication agent skill as a final technical gate. |
| Agent Skills/README.md | Introduces the Agent Skills index and documents a portability-focused authoring format. |
| Agent Skills/pre-publication-check/SKILL.md | Adds the detailed pre-publication audit skill, including checks and a standardized reporting template. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Section 1 (now 'Licensing and third-party code'): scan the complete codebase (and history) for copied/inlined code; FAIL on Apache-incompatible copied code and on copied code without attribution / missing NOTICE. - Add mandatory security-audit section (section 9) with hand-off to a planned dedicated security-audit skill; absence is a FAIL. - Add optional post-publication recommendation to mint a Zenodo DOI. - Add re-run modes (initial / follow-up after fixes / periodic ~12-month re-audit) that read the previous report and track prior findings. - Report format is Markdown with run type, previous-report reference, status of previous findings, recommendations and next-review date. - Clarify reports are retained but never stored in the public repo; a designated access-controlled store is to be provided.
- ADR-009 (Proposed): record the decision to store audit reports in the private ecmwf/repo-audits repo (access = org/Enterprise owners), with layout audits/<org>/<repo>/, UTC-timestamped filenames, YAML front-matter incl. audited commit SHA, secret-value redaction, and retention. Add index row. - SKILL.md: point 'Report storage' and the follow-up run mode at ecmwf/repo-audits; specify the path/filename/front-matter and redaction. - Open-Sourcing-Software.md: file reports in the private store and gate publication on READY audits; link ADR-009.
…ccess may come later
- New Agent Skills/security-audit/SKILL.md: risk-tiered security audit (report + gate, owner fixes). Threat model + attack-class taxonomy (generalised tensogram A-L plus conditional ML and LLM/agent classes), CRITICAL/HIGH-blocking severity model, automated tooling sweep (SAST, osv-scanner/trivy/grype+syft SBOM, per-ecosystem scanners, zizmor, scorecard), manual sensitive-surface review (deserialization, injection, memory safety/FFI, crypto, ML loading), supply chain & CI/CD, OpenSSF-mapped posture, and a high-risk deep dive (adversarial tests + bounded fuzzing + sanitizers/miri). SECURITY.md is a non-blocking recommendation pointing disclosures to https://support.ecmwf.int. Reports filed in ecmwf/repo-audits. - pre-publication-check SKILL.md section 9: convert from 'planned/interim' to a hand-off that runs the security-audit skill; missing audit or open CRITICAL/HIGH is a FAIL. - Agent Skills/README.md: index the new skill.
The audit is not only a one-off pre-publication gate: it can be re-run any time (initial / follow-up / periodic) to confirm an already-public repository still complies. Rename the skill to reflect that ongoing-compliance role and to align its name with the Open-Source-Audit report type it produces. - Move Agent Skills/pre-publication-check/ -> Agent Skills/open-source-audit/; update frontmatter name, title, intro, run-mode wording and report header. - Update all cross-references: Agent Skills/README.md, security-audit/SKILL.md, Legal/Open-Sourcing-Software.md, ADR-009 (also drop now-inaccurate 'planned' wording for security-audit; bump Last Updated). - Report/audit type token (Open-Source-Audit) and audit_type (open-source) are unchanged.
tlmquintino
changed the title
- Repository Structure/SECURITY.md: language- and project-agnostic security policy that routes vulnerability reports to the ECMWF Support Portal (https://support.ecmwf.int); drop-in for any ECMWF repo. Based on ECMWF Tensogram's SECURITY.md, with module/language-specific scope removed. - Repository Structure/README.md: document that each repo must ship a SECURITY.md and copy this template. - security-audit skill: point the SECURITY.md posture recommendation at the template.
- open-source-audit: use portable 'grep -riE' with an explicit search root instead of GNU-only '\|' alternation in the GPL-reference check. - open-source-audit: cite Guidelines/External-Contributions.md for the clean SemVer 'x.y.z' (no 'v' prefix) production-tag convention. - open-source-audit: include the required YAML front-matter in the report format so it is self-consistent with 'Report storage', the repo-audits SCHEMA, and ADR-009. - Open-Sourcing-Software.md: the security audit now exists and is required — drop the 'once available' wording so both audits must be READY.
open-source-audit: - Add Blocker vs Advisory classification: verdict is NOT_READY only for true publication blockers (secrets, licence incompatibility, missing licence, IPR/provenance, NOT_READY security audit); low-impact hygiene is advisory and does not block. Report format gains an 'Advisory' section; fail_count counts blockers only. - Header check: scan git-tracked files (git ls-files); don't flag generated artefacts like setuptools_scm _version.py. - Clarify reuse/scancode are provenance leads, not a gate; REUSE/SPDX is optional at ECMWF (prose Apache header is the standard). security-audit: - State that the verdict follows the auditor's TRIAGED severity, not the raw tool label (e.g. zizmor rating an unpinned first-party reusable workflow 'high'). - High-risk trigger clarified to mean repo-owned parser/native code; using an audited upstream lib to read data is a review surface, not a deep-dive trigger. - Note semgrep --config auto needs metrics on (offer p/... packs); add nbqa guidance for notebooks (linter must live in nbqa's env).
|
An alternative is to use more of GitHub's built-in security reporting features. It still suggests that audits go to a central repo but I question even that. Maybe everything could just be GHSAs. It should be less admin for org-owners. Claude's concept: Consider leaning on GitHub-native security features (and possibly shrinking/replacing the central store)Broadly +1 on the framework. One thing I'd like us to weigh before we lock in the 1. The native disclosure pipeline: PVR → advisory → private fork → CVE → DependabotGitHub's mechanism has three layers, and right now the PR only addresses the policy layer:
Why this may work better than a Markdown file in a separate repo, specifically for security findings:
Suggested change: have the 2. Org-level
|
Weak-copyleft deps (MPL-2.0, EPL) are file-level and Apache-compatible — do not fail them. An UNKNOWN licence usually means missing metadata, not a bad licence; confirm manually rather than auto-failing.
GitHub's hosted security features (Dependabot, code scanning, secret scanning, push protection) are orthogonal to these skills — whether ECMWF adopts them is a separate decision. Reword the skills so they neither depend on nor recommend enabling those services; the audits run their own scans (gitleaks, semgrep, pip-audit, osv-scanner, trivy, …) regardless. - security-audit: replace 'code scanning / secret scanning + push protection / Dependabot alerts enabled' posture checks and the 'enable CodeQL/Dependabot/ secret scanning' follow-up with a tool-agnostic 'CI runs SAST + dependency scanning' item; generalise the dependency-update check. - open-source-audit: generalise the CI security-scanning best-practice item. - ADR-009: drop the GitHub push-protection reference in the mitigation note. - Repository Structure: reconcile SECURITY.md wording with the skill — 'should contain (recommended)', not 'must … verbatim', consistent with the non-blocking recommendation.
|
Thanks for the very thorough write-up, James. To keep this PR focused: GitHub's own security services — private vulnerability reporting, security advisories/GHSAs, Dependabot, code scanning, secret scanning/push protection — are orthogonal to these skills and out of scope here. Whether we adopt them is a separate decision we can take independently; the skills shouldn't depend on, assume, or recommend them. So I've made the skills agnostic to those services: they run their own scans (gitleaks, semgrep, bandit, pip-audit, osv-scanner, trivy, scorecard, …) regardless of what's enabled on the repo. I removed the posture checks and follow-ups that assumed native code scanning / secret scanning / Dependabot were enabled, and replaced them with tool-agnostic "CI runs SAST + dependency scanning" items. I did take the one concrete consistency point you raised: The larger ideas — advisories as the home for vulnerabilities, an org-default |
Coordinated Vulnerability Disclosure workflow implementing ADR-011: PVR as the primary private reporting channel on public repos (Support Portal / SECURITY.md as fallback), triage -> fix in a draft advisory + temporary private fork -> coordinated release -> publish with CVE and reporter credit. Includes scope (ecmwf, ecmwf-ifs, ecmwf-training), enabling/notification requirements, roles, response targets (5/10 business days, 90-day remediation), a maintainer checklist, and a note distinguishing reported vulnerabilities (advisories) from audit reports (repo-audits store). Indexed in Guidelines/README.md and the root README; the Open Source Principles security sentence now links to it.
…o PVR-first - Repository Structure/SECURITY.md: route reporters to GitHub private vulnerability reporting first (Security tab -> Report a vulnerability), with the Support Portal as fallback; response promises unchanged (5/10 business days). - Repository Structure/README.md: Security Policy section notes the PVR-first routing and links the Security Vulnerability Disclosure procedure. - security-audit skill: SECURITY.md posture item now reflects PVR-first per the procedure; new advisory (non-blocking) posture check that PVR is enabled on public repos; follow-ups bullet updated; for already-public repos, confirmed CRITICAL/HIGH findings should be tracked as draft security advisories per the procedure (skill remains report-only).
|
Follow-up: by direction of the repo owner, this PR now adopts the PVR/advisories part of your proposal for vulnerability disclosure — see ADR-011 and the new Unchanged: GitHub's security scanning services remain out of the skills' scope, and audit reports stay in the |
'syft . -o spdx-json && grype sbom:-' left grype waiting on empty stdin; use a pipe instead.
- SECURITY.md template: PVR reports are visible to repository administrators and security managers, matching the disclosure procedure's wording. - security-audit: 'syft . -o spdx-json | grype sbom:-' — pipe plus explicit stdin directive so the SBOM example runs as written.
- REUSE.toml template: add <YEAR> to both SPDX-FileCopyrightText annotations, consistent with the SPDX/REUSE standard's year requirement. - CITATION.cff: use explicit <x.y.z> / <YYYY-MM-DD> placeholders so adopters cannot accidentally ship the template defaults.
Revise the access model: the store is writable by GitHub Enterprise / organisation owners, who serve as the auditors, and readable only internally at ECMWF (never external). Repository admins can therefore consult their own repository's reports without running audits. Ripple the wording through the open-source-audit skill's Report storage section and the open-sourcing checklist.
…h root Without a path operand, non-GNU grep reads stdin; add '.' so the examples scan the tree when copy/pasted (matching the earlier GPL-grep fix).
ADR-009 was taken by the Repository Audit Store (merged via #68), so this ADR is renumbered to ADR-012. Resolve the ADR/README.md index conflict (keep 009-011, add ADR-012), rename ADR-009-JSON-Support.md -> ADR-012-JSON-Support.md, fix the index link (previously pointed at a non-existent ADR-009-JSON-Schema.md), and correct the title (number + 'Arcitectural' typo).
Summary
Introduces an Agent Skills area to the Codex and the governance plumbing around it: two reusable, model-agnostic AI-agent skills that audit ECMWF repositories, a decision record for where their reports are stored, and the wiring into the existing open-sourcing process.
This started as a single "pre-publication check" skill and grew (through review) into a small, coherent audit framework.
What's included
1.
Agent Skills/frameworkAgent Skills/section, linked from the rootREADME.md.2.
open-source-auditskill (waspre-publication-check)A technical open-source compliance audit, run before a repo is made public and re-runnable any time to confirm continued compliance. Renamed from
pre-publication-checkto reflect that ongoing role and to align with theOpen-Source-Auditreport type it produces.3.
security-auditskillThe concrete implementation of the mandatory security step, risk-tiered and report-only (it gates but never fixes code or flips visibility).
SECURITY_ANALYSIS.md) plus conditional ML-model-loading and LLM/agent classes.osv-scanner/trivy/grype+syftSBOM, per-ecosystem scanners,zizmor, OpenSSFscorecard), manual review of security-sensitive surfaces, supply-chain/CI checks and posture; high-risk repos additionally get a deep dive (adversarial tests + bounded fuzzing + sanitizers/miri).SECURITY.mdis a non-blocking recommendation pointing disclosures to https://support.ecmwf.int.4.
ADR-009— Repository Audit Store (Accepted)Records the decision to store all audit reports in a dedicated private repo,
ecmwf/repo-audits:audits/<org>/<repo>/<YYYY-MM-DDThhmm>-<Type>.md(UTC), coveringecmwf,ecmwf-ifs,ecmwf-training.5. Open-sourcing process wiring
Legal/Open-Sourcing-Software.md: run theopen-source-auditskill as the final technical gate, file the report inecmwf/repo-audits, and gate publication onREADYaudits.Companion repository (not in this PR)
The private store
ecmwf/repo-auditshas been created and scaffolded separately (README,SCHEMA.md,CODEOWNERS, report templates, per-org folders, a worked example; secret-scanning + push protection on;mainprotected). This PR contains only the Codex-side documentation and skills.Notes for reviewers
Accepted.add-pre-publication-agent-skill) is legacy — the skill is nowopen-source-audit; the branch was left as-is to preserve this PR.Open-Source-Audit,Security-Audit) andaudit_typevalues are stable across the rename.SPDX / REUSE licence headers — ADR-010 (added to this PR)
This PR also adds ADR-010, which adopts SPDX short-form licence identifiers and REUSE-compliant file headers as the ECMWF-wide standard going forward, plus the supporting standard and copy-ready templates. Status is Accepted — the standard is in force (new files must use the SPDX/REUSE header; existing files migrate incrementally). A copy-ready
CITATION.cffexample was also added toRepository Structure/.Why: ECMWF publishes ~160 repositories under Apache 2.0, and its libraries are routinely copied, vendored, and forked downstream. Today licensing lives in hand-maintained, multi-line boilerplate that is inconsistent across the estate, hard for tools to validate, and not machine-identifiable when a single file leaves its repository — and we have no organisation-wide, machine-readable metadata from which to generate an SBOM. SPDX is an ISO standard (ISO/IEC 5962:2021) and an SBOM format named in EU CRA guidance.
What the standard requires: each ECMWF-authored, commentable file carries two tags —
Additional copyright holders (e.g. partner-co-developed code —
Crown Copyright, Met Office) get an extraSPDX-FileCopyrightText:line. The unmodified Apache text lives inLICENSES/Apache-2.0.txt; a top-levelLICENSEholds the Apache text with ECMWF's intergovernmental notice at its tail;NOTICErecords ECMWF copyright and that notice plus any third-party attributions; non-commentable files are covered byREUSE.toml. Third-party files keep their own headers. Adoption is incremental.Advantages: precise and machine-readable; the licence and copyright travel with each file downstream; SBOM-ready (
reuse spdx); consistent and easy to bulk-maintain; verifiable viareuse lint; lighter, uniform headers.Disadvantages / costs: a one-off migration per repository (automatable with
reuse annotate); the intergovernmental notice is no longer repeated in every file (preserved inLICENSEandNOTICE); new required artefacts per repository (LICENSES/,NOTICE, and where neededREUSE.toml); ongoing discipline to prevent drift.Enforcement: pre-commit hooks and CI checks are advised, not mandated; templates for both are included.
Files:
ADR/ADR-010-SPDX-License-Identifiers-and-REUSE-Headers.md,ADR/README.md(index),Legal/SPDX-and-REUSE.md(standard),Legal/SPDX-REUSE-templates/(NOTICE, LICENSE tail, REUSE.toml, pre-commit, CI). Theopen-source-auditskill now accepts either the SPDX or the legacy prose header and recommends SPDX going forward.Coordinated Vulnerability Disclosure — ADR-011 + procedure (added to this PR)
Adds ECMWF's coordinated vulnerability disclosure mechanism and workflow, built on GitHub Private Vulnerability Reporting (PVR) and repository security advisories:
ADR/ADR-011-Coordinated-Vulnerability-Disclosure-via-PVR.md(Accepted) — records the decision: PVR + advisories are the primary disclosure channel for public repos acrossecmwf,ecmwf-ifs,ecmwf-training; Support Portal remains the fallback and the private-repo channel. Distinguishes externally reported vulnerabilities (→ private advisories) from audit reports (→ecmwf/repo-audits, ADR-009).Guidelines/Security-Vulnerability-Disclosure.md— the end-to-end procedure: private report (PVR primary,SECURITY.md/portal fallback) → triage → fix in a draft advisory + temporary private fork (never on public branches) → coordinated release/embargo → publish with optional CVE (GitHub is a CNA), downstream notification, and reporter credit. Includes enabling/notification requirements, roles (Maintainer, Technical Officer, Head of Development, security managers), and a maintainer checklist. Indexed inGuidelines/README.mdand the root README; the Open Source Principles security sentence now links to it.SECURITY.md— the template now routes reporters to the repository's Security → Report a vulnerability button first, Support Portal as fallback; thesecurity-auditskill gains an advisory (non-blocking) posture check that PVR is enabled, and recommends tracking confirmed CRITICAL/HIGH findings on already-public repos as draft advisories.Response targets: acknowledge within 5 business days, initial assessment within 10 business days (matching the
SECURITY.mdtemplate), remediation target 90 days (shorter if actively exploited).Note on scope: GitHub's security scanning services (code scanning, secret scanning, dependency alerts) remain out of scope for the audit skills — PVR/advisories are a disclosure channel, not a scan, so this does not conflict with that stance.
Commit history
repo-auditsstore (owners-only while maturing).security-auditskill and convert the security step into a hand-off.pre-publication-check→open-source-audit(ongoing-compliance framing).