Skip to content

Disclose AI assistance and record human verification #14

Open

Description

@mmcky

Model descriptions and implementations in the library may be drafted with help from AI. This issue adds two indicators to each one: how much AI was involved in producing it, and whether a person has verified it. Both describe process. Neither claims that a document is free of errors, and neither ranks documents.

AI assistance: a level from 0 to 4, shown as a four-segment bar.

Level Meaning
0/4 No AI was used in the text or code.
1/4 AI-assisted editing: people wrote every passage, and AI suggested wording, grammar or consistency fixes.
2/4 AI-assisted drafting: people wrote most of it; AI drafted some passages, which people rewrote.
3/4 AI-drafted, human-directed: AI drafted most of it to an outline and decisions from people, who edited it throughout.
4/4 AI-drafted with light human editing.

Three rules keep the levels consistent:

  • The level describes how the text and code were produced. Choices about the model, the calibration and the results always belong to the authors.
  • Report the highest level that applies to any substantial part, such as one section or the code, so one AI-drafted section isn't averaged away.
  • List what AI was used for, from a fixed list: drafting, editing, code, derivation checks, literature, figures.

Authors state the level in their pull request, and the reviewer confirms it as part of the review process from #6.

Human verified means a named person has read the document end to end and checked it against its sources. It is recorded with the person's name, the date and the version they verified. When a new version comes out, the indicator shows "not yet verified" until someone verifies that version. This is worked out from the recorded version, so nobody has to remember to clear it.

Display

  • The AI bar uses one neutral colour and varies only how many segments are filled. A red-to-green scale would read as "more AI means less trustworthy", which is the quality judgement these indicators avoid. The bar also has to survive greyscale printing.
  • Verification follows the site's existing slot style: a solid blue border for verified for this version, a dashed grey one for not yet verified. Both always carry text, so neither relies on colour alone.
  • Both indicators appear in the PDF's version box from Version each model description and let implementations cite the version they follow #13, on each description's web page from Publish each model description as a web page alongside its PDF #12, and in each slot of the site's library table. Each links its level to the definitions.

Where it is recorded

What Where
The definitions: levels, rules and display A section of a root CONTRIBUTING.md, next to the versioning rules from #13. Each model's submission guide links to it.
A model description's level and verification The front matter of models/<model>/model-description.md: ai.level, ai.used_for, verified.by, verified.date and verified.version.
An implementation's level and verification projects/<toolkit>/<model>/metadata.yml, with the same keys.

The values live in the files they describe, so every change to them is reviewed, and each version's tag preserves that version's values.

Proposal: make a minimal metadata.yml required. #5 makes the reported numbers required but the files optional. For the site to show each implementation's version, AI level and verification automatically, those fields need a fixed place. Every implementation would include a metadata.yml with model_description_version, authors, ai and verified, and every other field would stay optional. This revises design decision 3 of #5 and the submission guide, so it needs agreement before it is built.

This is planned to land together with #12 and #13, after #5 merges.

Activity

  1. mmcky commented on Oct 20, 2026

    @mmcky
    ContributorAuthor

    Decided on 20 October 2026:

    • Every implementation includes a metadata.yml. The required fields are model_description_version, authors, ai and verified; every other field stays optional.
    • No filename convention. Versions do not go in file names: renaming a file at every version would break the site's links and any links from elsewhere. Stable paths, plus the version tags from Version each model description and let implementations cite the version they follow #13, pin the version an implementation follows.
    • Help for submitters, in the pull request for Publish each model description as a web page alongside its PDF #12–Disclose AI assistance and record human verification #14: a schema for metadata.yml in the repository, a CI check that names any missing or invalid field, and a copyable example in the submission guide.
    • Later, when outside submissions start: a "Submit an implementation" issue form whose answers a GitHub Action turns into a metadata.yml on the submitter's pull request. The site is static, so a GitHub form provides the web submission and sign-in without a server.
    • Several solutions to one model from one toolkit (for example EGM and value function iteration) would each get a subdirectory with its own metadata.yml. That rule will be written when the first such submission arrives.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions