Skip to content

Initialize SciCodes skills repo with experimental assess-scicodes-practices skill - #1

Merged
alee merged 10 commits into
mainfrom
copilot/assess-scicodes-practices
Oct 31, 2026
Merged

alee merged 10 commits into
mainfrom
copilot/assess-scicodes-practices

Conversation

Copilot AI commented Oct 31, 2026 •

Copy link
Copy Markdown

Adds an experimental assess-scicodes-practices skill that accepts a registry or repository URL and prepares an evidence-based report, adoption checklist, and structured JSON assessment for member review.

The rubric pins Garijo et al. (2022), Nine best practices for research software registries and repositories, PeerJ Computer Science 8:e1023 (DOI: 10.7717/peerj-cs.1023), using the supplied PDF. It contains 52 page-anchored recommendations across nine practices, with explicit assessment interpretations and applicability guidance. Four optional enhancements remain outside baseline scores.

The supplied Best Practices Tracking Scorecard establishes the practice headings and 0–4 scale plus N/A. Missing evidence remains unknown and prevents a definitive practice score. The historical crosswalk records column mappings, source provenance, and limits on comparing historical values. No overall member scores or rankings are produced.

The skill resolves collection URLs to the relevant service, distinguishes policy disclosure from verified implementation, and keeps findings as drafts. It does not modify member services or publish assessments. The original paper and member scorecard rows are not bundled.

Validation checks the schema and rubric, their vocabulary compatibility, criterion coverage, evidence references, baseline/version consistency, and derived scores. CI runs make check; tests use fictional resources on reserved domains. Score decisions live in the rubric's declarative rules.

Validation: make check passes, including lint, repository validation, and all 39 tests. The model evaluation harness has regression coverage, but no live-model or end-to-end browsing assessment has been conducted. Criterion grouping, equal weighting, and applicability remain experimental assessment interpretations pending SciCodes review.

Copilot AI changed the title [WIP] Initialize SciCodes skills with assess-scicodes-practices Initialize SciCodes skills repo with experimental assess-scicodes-practices skill Oct 31, 2026
Copilot AI requested a review from alee October 31, 2026 02:18
@alee
alee marked this pull request as ready for review October 31, 2026 03:37
@alee
alee force-pushed the copilot/assess-scicodes-practices branch from c512883 to 35650fc Compare October 31, 2026 04:46
Copilot AI and others added 3 commits October 30, 2026 22:12
Provide evidence-based assessment drafts with declarative rules, output templates, and local validation.

Co-authored-by: alee <22534+alee@users.noreply.github.com>
Reject malformed configuration and unsupported findings, verify assessment provenance, and separate validation phases. Clarify resource scope, exercise the evaluation interface, and document commit conventions.
Use the supplied scorecard and PeerJ paper to define the initial 0–4 rubric with 52 source-backed recommendations. Record source provenance and applicability while keeping optional guidance outside baseline scores.
@alee
alee force-pushed the copilot/assess-scicodes-practices branch from 35650fc to e0c80dc Compare October 31, 2026 05:13
Comment thread .github/workflows/check.yml Outdated
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pin external GitHub Actions to verified full commit SHAs, with release-version comments. Configure weekly Dependabot updates for the github-actions ecosystem. Reuse an existing workflow dependency lockfile if present.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinned actions/checkout (v7.0.1, 3d3c42e) and actions/setup-python (v7.1.0, 6d4fd22) to full SHAs, resolved from the repositories' release tags via the GitHub API, with version comments. Added weekly Dependabot for github-actions. There was no existing workflow lockfile to reuse. Note this moves both actions from v4/v5 to the latest majors.

Comment thread README.md

Assumptions: the agent can browse public web pages (otherwise it assesses only evidence you supply and says current public evidence was not verified) and can write to a working directory (otherwise it returns the three outputs inline).

### Installation

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reference https://agentskills.io/home instead as the canonical spec and https://www.skills.sh/ as an updated resource

reference skill installation details https://agentskills.io/client-implementation/adding-skills-support

npx skills add | update should be covered as well as project-specific or global skills? global skills can be annoying if they have a certain way of organizing information and default to it for tasks that don't need it like omfa/b or the fair skill making manifests and tracking data as if everything is a computational model.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the three references and a note to prefer project-specific over global installs. I could not reach agentskills.io or skills.sh from this environment, so I left out npx skills add|update and the global-versus-project details rather than guess. Could you paste the relevant doc excerpts, or allow those domains? I'll then document the commands and scopes.

Co-authored-by: alee <22534+alee@users.noreply.github.com>
Co-authored-by: alee <22534+alee@users.noreply.github.com>
Co-authored-by: alee <22534+alee@users.noreply.github.com>
@alee
alee merged commit 1991509 into main Oct 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants