Repository navigation
Initialize SciCodes skills repo with experimental assess-scicodes-practices skill - #1
Conversation
assess-scicodes-practices skill
c512883 to
35650fc
Compare
Provide evidence-based assessment drafts with declarative rules, output templates, and local validation. Co-authored-by: alee <22534+alee@users.noreply.github.com>
Reject malformed configuration and unsupported findings, verify assessment provenance, and separate validation phases. Clarify resource scope, exercise the evaluation interface, and document commit conventions.
Use the supplied scorecard and PeerJ paper to define the initial 0–4 rubric with 52 source-backed recommendations. Record source provenance and applicability while keeping optional guidance outside baseline scores.
35650fc to
e0c80dc
Compare
| check: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 |
There was a problem hiding this comment.
Pin external GitHub Actions to verified full commit SHAs, with release-version comments. Configure weekly Dependabot updates for the github-actions ecosystem. Reuse an existing workflow dependency lockfile if present.
There was a problem hiding this comment.
Pinned actions/checkout (v7.0.1, 3d3c42e) and actions/setup-python (v7.1.0, 6d4fd22) to full SHAs, resolved from the repositories' release tags via the GitHub API, with version comments. Added weekly Dependabot for github-actions. There was no existing workflow lockfile to reuse. Note this moves both actions from v4/v5 to the latest majors.
|
|
||
| Assumptions: the agent can browse public web pages (otherwise it assesses only evidence you supply and says current public evidence was not verified) and can write to a working directory (otherwise it returns the three outputs inline). | ||
|
|
||
| ### Installation |
There was a problem hiding this comment.
Reference https://agentskills.io/home instead as the canonical spec and https://www.skills.sh/ as an updated resource
reference skill installation details https://agentskills.io/client-implementation/adding-skills-support
npx skills add | update should be covered as well as project-specific or global skills? global skills can be annoying if they have a certain way of organizing information and default to it for tasks that don't need it like omfa/b or the fair skill making manifests and tracking data as if everything is a computational model.
There was a problem hiding this comment.
I added the three references and a note to prefer project-specific over global installs. I could not reach agentskills.io or skills.sh from this environment, so I left out npx skills add|update and the global-versus-project details rather than guess. Could you paste the relevant doc excerpts, or allow those domains? I'll then document the commands and scopes.
Co-authored-by: alee <22534+alee@users.noreply.github.com>
Co-authored-by: alee <22534+alee@users.noreply.github.com>
Co-authored-by: alee <22534+alee@users.noreply.github.com>
Adds an experimental
assess-scicodes-practicesskill that accepts a registry or repository URL and prepares an evidence-based report, adoption checklist, and structured JSON assessment for member review.The rubric pins Garijo et al. (2022), Nine best practices for research software registries and repositories, PeerJ Computer Science 8:e1023 (DOI: 10.7717/peerj-cs.1023), using the supplied PDF. It contains 52 page-anchored recommendations across nine practices, with explicit assessment interpretations and applicability guidance. Four optional enhancements remain outside baseline scores.
The supplied Best Practices Tracking Scorecard establishes the practice headings and 0–4 scale plus N/A. Missing evidence remains unknown and prevents a definitive practice score. The historical crosswalk records column mappings, source provenance, and limits on comparing historical values. No overall member scores or rankings are produced.
The skill resolves collection URLs to the relevant service, distinguishes policy disclosure from verified implementation, and keeps findings as drafts. It does not modify member services or publish assessments. The original paper and member scorecard rows are not bundled.
Validation checks the schema and rubric, their vocabulary compatibility, criterion coverage, evidence references, baseline/version consistency, and derived scores. CI runs
make check; tests use fictional resources on reserved domains. Score decisions live in the rubric's declarative rules.Validation:
make checkpasses, including lint, repository validation, and all 39 tests. The model evaluation harness has regression coverage, but no live-model or end-to-end browsing assessment has been conducted. Criterion grouping, equal weighting, and applicability remain experimental assessment interpretations pending SciCodes review.