feat(ci): enforce recipe placement, support vertical skills, and fix silent validation gates - #2354
Merged
Conversation
…ate documentation
…tract Vertical skills use a vertical (retail/, hr/, finance/) where core/ and contrib/ use a language. The tooling had one concept for that path slot, so it read skills/retail/store-ops as recipe "skills/retail" — validating the vertical folder and never seeing the solution at all. - Teach the collectors that skills/ always namespaces by position, not by name (NAMESPACE_REQUIRED_ROOTS + is_namespace_path). - Add tools/validate_placement.py and a CI job to reject a solution placed directly under skills/, which the collectors otherwise skip silently. - Require EVAL.yaml, plus scripts/, assets/, references/ and tests/unit/ via a new required_dirs policy section (required_files cannot express a directory). Empty directories pass. - Match required names in Python rather than via is_file()/is_dir(): the filesystem answered case-insensitively on macOS, so a recipe could pass locally and fail on Linux CI. EVAL.yaml opts into leniency explicitly; files that other tools resolve by exact name must never do so. - Give skills/ the same size limits as contrib/, and a CODEOWNERS entry. Also settles the naming collision between the two things called "skills": repo skills (.agents/skills/, used to build this repo) and vertical skills (skills/, shipped to users). Updates the repo skills' own stale path references, and fixes a scaffold bug where contrib/python — the documented home for new Python recipes — was rejected by the CLI allow-list.
Three failures on PR #2354: - zizmor `unpinned-uses` (12 findings, a mandatory check). The new jobs used tag refs, and touching two sibling workflows pulled their existing tag refs into the changed-file scan. All 12 now use the SHAs already canonical elsewhere in the repo. - dependabot.yml was missing an entry for contrib/python/market-research-agent, which landed on main while this branch was open. Merged main and regenerated: 16 entries. - core/python/ambient-expense-agent/frontend/uv.lock was stale. Pre-existing, surfaced because the python-dependency-policy.yml change on this branch triggers a full scan rather than a diff-scoped one. Regenerated; the only delta is registry URL normalisation, no version changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three related fixes to recipe validation. Each closes a case where CI either said nothing, said something false, or checked the wrong directory.
1. Vertical skills —
skills/<vertical>/<solution>/Vertical skills use a vertical (
retail/,hr/,finance/) wherecore/andcontrib/use a language. The tooling had a single concept for that path slot, so it readskills/retail/store-opsas recipeskills/retail— validating the vertical folder and never seeing the solution at all.NAMESPACE_REQUIRED_ROOTS+is_namespace_path()teach the collectors thatskills/namespaces by position, not by nametools/validate_placement.py+check-recipe-placementjob rejects a solution dropped directly underskills/, which the collectors otherwise skip silentlyEVAL.yamladded to required files;scripts/,assets/,references/,tests/unit/required via a newrequired_dirspolicy section, sincerequired_filescannot express a directory. Empty directories pass.skills/gets the same size limits ascontrib/, plus a CODEOWNERS entryValidated against a real skill (
retail-product-search): the required shape matches what it already ships, and the checks correctly flag its 5 missing pieces.2. Retired legacy folders
python/agents/,java/agents/,go/agents/,kotlin/agents/,typescript/agents/are closed. A PR that adds or modifies anything there now fails with the destination spelled out:Deletions and renames are exempt (
--diff-filter=AM), so migrating a recipe out is never blocked. Bots are exempt; maintainers can override with anallow-legacy-pathlabel. The five legacy READMEs get a deprecation banner — they currently read as live, welcoming guides, which is why the PRs keep arriving.Heads-up: 53 open PRs will go red on merge. That's the intent — they currently wait days for a manual "wrong folder" reply.
3. Dependabot gate told the truth
The generated
dependabot.ymlclaimed it was "regenerated automatically on every push to main". It isn't — the sync workflow isworkflow_dispatch-only, and its own comments explain why. The header now states the real contract, and--checkprints a diff of the missing entry instead of a dead end.Also fixed along the way
is_file()is case-insensitive on macOS, soPyProject.tomlpassed locally and failed on Linux CI. Names are now compared in Python.EVAL.yamlopts into leniency explicitly; files other tools resolve by exact name (pyproject.toml,uv.lock,manifest.yaml,tests/test_runnability.py) never can.scaffold-python-reciperejectedcontrib/python— the documented home for new Python recipes — because of a stale CLI allow-list..agents/skills/, build this repo) vs vertical skills (skills/, shipped to users). Both now defined in the handbook glossary.Verification
mainruff check/ruff formatclean acrosstools/and.github/scripts/uv lock --checkin syncgit mvof a recipe out of a retired folder producesR100and passesAfter merge
Two things CI can't do itself:
Known gaps
skills/untilcore/rag-agent-searchandcore/rag-vector-searchare deleted; widening it afterwards is a one-line changemanifest.yamlis still invisibleassets/design-spec.mdshould be a required file is still open