Skip to content

ci-secrets.md

Latest commit

 

History

History
379 lines (311 loc) · 16.6 KB

ci-secrets.md

File metadata and controls

379 lines (311 loc) · 16.6 KB
Raw

CI secrets and access control for leanprover/lean-eval-submissions

This document is the source of truth for the repository-level GitHub Apps, personal access token, and branch-protection settings described below. It does not inventory the complete runtime authentication posture. The current Worker, GitHub-environment, deploy-key, OAuth, and AWS credential boundaries are in INFRASTRUCTURE.md. Read both documents before changing the authentication posture; neither records secret values.

The benchmark repository leanprover/lean-eval has its own docs/ci-secrets.md covering its lean-eval-regenerator App; that one is unrelated to this pipeline.

Audit checklist

Item Type Stored as Used by
lean-eval-bot GitHub App LEAN_EVAL_BOT_CLIENT_ID, LEAN_EVAL_BOT_PRIVATE_KEY submission.yml (fetch); protected deploy-worker.yml (broker-secret convergence)
lean-eval-recorder GitHub App LEAN_EVAL_RECORDER_CLIENT_ID, LEAN_EVAL_RECORDER_PRIVATE_KEY submission.yml (record)
lean-eval-archiver GitHub App LEAN_EVAL_ARCHIVER_CLIENT_ID, LEAN_EVAL_ARCHIVER_PRIVATE_KEY submission.yml (archive)
LEADERBOARD_WRITE_TOKEN Fine-grained PAT LEADERBOARD_WRITE_TOKEN submission.yml (leaderboard redeploy dispatch)
Ruleset main protection Repository Ruleset (config in this file, applied via API) branch protection on main
Ruleset Protect staging Results Repository Ruleset (config in this file, applied via API) branch protection on staging-results

The three Apps in this document are owned by the personal account kim-em; Kim Morrison is their current credential custodian. For any App private-key rotation, generate one replacement key while the old key remains valid, replace only that App's *_PRIVATE_KEY repository secret, verify the matching protected workflow path with intake disabled or in staging, and then delete the old key in the App settings. Immediate revocation is deletion of the affected App key followed by removal or replacement of its repository secret. Do not reuse one App's key for another App. No alternate App custodian is currently recorded. No current key expiry, key ID, or fingerprint is recorded: GitHub's repository-secret and public App APIs do not expose which App key a stored private-key secret contains. Do not infer those facts from a repository-secret creation or update timestamp.

To check the live state at any time:

gh secret list -R leanprover/lean-eval-submissions
for app in lean-eval-bot lean-eval-recorder lean-eval-archiver; do
  gh api "/apps/$app" --jq '{slug, id, client_id}'
done
gh api /repos/leanprover/lean-eval-submissions/rules/branches/main --jq '[.[].type]'

GitHub App: lean-eval-bot

Used by .github/workflows/submission.yml to mint installation tokens that fetch submission source from contributor repositories (which may be private).

App settings

  • Owner account: kim-em (User account).
  • App ID: 3346375 (public identifier used for installation metadata).
  • Webhook: deactivated.
  • Repository permissions:
    • Contents: Read
  • Where can this GitHub App be installed: Any account. Contributors install it on their own submission repos so the workflow can clone them.

Repository secrets (in leanprover/lean-eval-submissions)

  • LEAN_EVAL_BOT_CLIENT_ID — the app's Client ID. It is public, but stored as a secret to keep the workflow's app inputs together.
  • LEAN_EVAL_BOT_PRIVATE_KEY — the full PEM contents of a private key generated for the app.

Where used

.github/workflows/submission.yml, in the Mint lean-eval-bot installation token step, via actions/create-github-app-token. The minted token is scoped to the single Fetch submission step and is used to clone the contributor's submission source.

Protected .github/workflows/deploy-worker.yml also pipes these existing secrets directly into each private broker as LEGACY_SOURCE_APP_ID and LEGACY_SOURCE_APP_PRIVATE_KEY. That broker mints a separate repository-scoped contents/metadata-read token only to prove the exact repository and commit before server intake mutates State.

The issue template .github/ISSUE_TEMPLATE/submit.yml instructs contributors to install this app on their submission repo.

Reconstruction from scratch

This is the same App that previously served leanprover/lean-eval's submission workflow. The migration step is to install it on leanprover/lean-eval-submissions and copy its secrets here. A from-scratch rebuild:

  1. As the desired app owner, visit https://github.com/settings/apps/new.
  2. Fill in:
    • Name: lean-eval-bot
    • Homepage URL: https://github.com/leanprover/lean-eval-submissions
    • Webhook → Active: unchecked
    • Repository permissions → Contents: Read
    • Where can this GitHub App be installed: Any account
  3. Save → record both the App ID and Client ID.
  4. Generate a private key, download the .pem.
  5. Install the app on leanprover/lean-eval-submissions (so the workflow has an installation to mint tokens against).
  6. Set the secrets: 
    gh secret set LEAN_EVAL_BOT_CLIENT_ID -R leanprover/lean-eval-submissions --body <CLIENT_ID>
gh secret set LEAN_EVAL_BOT_PRIVATE_KEY -R leanprover/lean-eval-submissions < path/to/key.pem

GitHub App: lean-eval-recorder

Used by .github/workflows/submission.yml to push the record: commit (a results-store update) directly to this repo's main or isolated staging-results branch, bypassing the matching branch ruleset.

App settings

  • Owner account: kim-em (User account).
  • App ID: 3769615 (public identifier used by the ruleset bypass actor).
  • Webhook: deactivated.
  • Repository permissions:
    • Contents: Read and write
  • Where can this GitHub App be installed: Any account. (Required so the org can install it; the only intended installation is on leanprover/lean-eval-submissions itself.)
  • Installed on: leanprover/lean-eval-submissions only (single-repo installation).

This is a distinct App from lean-eval-bot, on purpose: lean-eval-bot is installed on arbitrary contributor repositories, so it must stay Contents: Read only. A write-capable App must never be installable on third-party repos.

Repository secrets (in leanprover/lean-eval-submissions)

  • LEAN_EVAL_RECORDER_CLIENT_ID — the app's Client ID. It is public, but stored as a secret to keep the workflow's app inputs together.
  • LEAN_EVAL_RECORDER_PRIVATE_KEY — the full PEM contents of a private key generated for the app.

Where used

.github/workflows/submission.yml, in the Mint lean-eval-recorder installation token step. The token authenticates the results-store/ checkout so the push-retry loop's git push origin HEAD:main lands on protected main.

Why an app and not GITHUB_TOKEN

GITHUB_TOKEN-authored pushes cannot bypass branch protection (no actor can put github-actions[bot] itself in the bypass list at workflow granularity). A dedicated app's principal can be in the bypass list (see the Ruleset section below); the bypass is then narrow because only this workflow has the app's secrets.

Reconstruction from scratch

  1. As the desired app owner, visit https://github.com/settings/apps/new.
  2. Fill in:
    • Name: lean-eval-recorder
    • Homepage URL: https://github.com/leanprover/lean-eval-submissions
    • Webhook → Active: unchecked
    • Repository permissions → Contents: Read and write (everything else stays "No access")
    • Where can this GitHub App be installed: Any account
  3. Save → record both the App ID (for the ruleset bypass actor) and the Client ID (for token minting).
  4. Generate a private key, download the .pem.
  5. Install the app on leanprover/lean-eval-submissions only.
  6. Set the secrets: 
    gh secret set LEAN_EVAL_RECORDER_CLIENT_ID -R leanprover/lean-eval-submissions --body <CLIENT_ID>
gh secret set LEAN_EVAL_RECORDER_PRIVATE_KEY -R leanprover/lean-eval-submissions < path/to/key.pem
  7. Add the App ID to the main and Protect staging Results rulesets' bypass lists (see Ruleset section below).

GitHub App: lean-eval-archiver

Used by .github/workflows/submission.yml to push age-encrypted submission tarballs and unencrypted metadata sidecars to leanprover/lean-eval-audit. See docs/audit-archive.md for the design.

App settings

  • Owner account: kim-em (User account).
  • App ID: 3856297 (public identifier used for installation metadata).
  • Webhook: deactivated.
  • Repository permissions:
    • Contents: Read and write
  • Where can this GitHub App be installed: Only on this account.
  • Installed on: leanprover/lean-eval-audit only (single-repo installation, scoped via repositories: lean-eval-audit in the workflow's actions/create-github-app-token step).

This is a distinct App from lean-eval-bot and lean-eval-recorder, on purpose:

  • lean-eval-bot is installed on arbitrary contributor repositories so it must stay Contents: Read only — bumping it to write would silently expand the trust boundary for every submitter who installed it.
  • lean-eval-recorder writes the leaderboard results store, a public repo; the audit archive is a private repo with a different access policy. Keeping the apps separate means an audit-archive compromise cannot also rewrite the leaderboard, and vice versa.

Repository secrets (in leanprover/lean-eval-submissions)

  • LEAN_EVAL_ARCHIVER_CLIENT_ID — the app's Client ID. It is public, but stored as a secret to keep the workflow's app inputs together.
  • LEAN_EVAL_ARCHIVER_PRIVATE_KEY — the full PEM contents of a private key generated for the app.

Where used

.github/workflows/submission.yml, in the Mint lean-eval-archiver installation token step of the archive job. The minted token authenticates the scripts/archive_submission.py push invocation, which writes one ciphertext file and one sidecar JSON to lean-eval-audit via the GitHub Contents API.

The archive job runs on a separate runner from the one that elaborates untrusted Lean (the evaluate job), so the write-capable installation token is never co-resident with an attacker-influenced process. See docs/audit-archive.md > "Threat model".

Reconstruction from scratch

  1. As the desired app owner, visit https://github.com/settings/apps/new.
  2. Fill in:
    • Name: lean-eval-archiver
    • Homepage URL: https://github.com/leanprover/lean-eval-audit
    • Webhook → Active: unchecked
    • Repository permissions → Contents: Read and write (everything else stays "No access")
    • Where can this GitHub App be installed: Only on this account
  3. Save → record both the App ID and Client ID.
  4. Generate a private key, download the .pem.
  5. Install the app on leanprover/lean-eval-audit only.
  6. Set the secrets: 
    gh secret set LEAN_EVAL_ARCHIVER_CLIENT_ID -R leanprover/lean-eval-submissions --body <CLIENT_ID>
gh secret set LEAN_EVAL_ARCHIVER_PRIVATE_KEY -R leanprover/lean-eval-submissions < path/to/key.pem

PAT: LEADERBOARD_WRITE_TOKEN

Fine-grained Personal Access Token used by .github/workflows/submission.yml to fire a repository_dispatch (event_type: results-advanced) at https://github.com/leanprover/lean-eval-leaderboard after a result is recorded, so the leaderboard site redeploys with the new result.

This is the same PAT leanprover/lean-eval and leanprover/lean-eval-leaderboard already use for their leaderboard interactions; the submission pipeline just needs its own copy of the secret.

Repository secrets

  • LEADERBOARD_WRITE_TOKEN in leanprover/lean-eval-submissions.

The token must be a fine-grained PAT with leanprover/lean-eval-leaderboard selected and Contents: Read and write — that permission is what the repository_dispatch REST endpoint authorizes on (per https://docs.github.com/en/rest/repos/repos#create-a-repository-dispatch-event).

Reconstruction from scratch

  1. Open https://github.com/settings/personal-access-tokens/new.
  2. Resource owner: leanprover (requires org-owner approval).
  3. Repository access: Only select repositories → leanprover/lean-eval-leaderboard.
  4. Repository permissions: Contents: Read and write.
  5. Save the token, then write it to this repo: 
    gh secret set LEADERBOARD_WRITE_TOKEN -R leanprover/lean-eval-submissions --body <TOKEN>

When rotating the PAT, update the copies in leanprover/lean-eval and leanprover/lean-eval-leaderboard together (see those repos' own docs). Replace all three copies before revoking the old token in its issuing GitHub account, then verify one dispatch from each caller. The repository APIs expose the secret names but not the PAT issuer or expiry; those two current facts are not recoverable from this repository and must not be guessed. Record them in the credential inventories during the next reviewed, packet-bound PAT rotation.

Branch protection on main (Repository Ruleset)

main is protected by a Repository Ruleset. The lean-eval-recorder app is on the bypass list so the record job can push results directly; everyone and everything else goes through a PR with a passing verify check.

Live state inspection

gh api /repos/leanprover/lean-eval-submissions/rulesets \
    --jq '.[] | {id, name, target, enforcement}'
gh api "/repos/leanprover/lean-eval-submissions/rulesets/<ID>" \
    --jq '{name, bypass_actors}'

Ruleset payload (canonical)

The <RECORDER_APP_ID> placeholder is the lean-eval-recorder App ID (currently 3769615, verified by the live-state command above). integration_id: 15368 pins the verify status check to the GitHub Actions app, so a hostile third-party app can't satisfy it with a bogus check of the same name.

{
  "name": "main protection",
  "target": "branch",
  "enforcement": "active",
  "conditions": { "ref_name": { "include": ["~DEFAULT_BRANCH"], "exclude": [] } },
  "rules": [
    { "type": "deletion" },
    { "type": "non_fast_forward" },
    { "type": "pull_request",
      "parameters": {
        "required_approving_review_count": 0,
        "dismiss_stale_reviews_on_push": false,
        "require_code_owner_review": false,
        "require_last_push_approval": false,
        "required_review_thread_resolution": false } },
    { "type": "required_status_checks",
      "parameters": {
        "strict_required_status_checks_policy": false,
        "required_status_checks": [ { "context": "verify", "integration_id": 15368 } ] } }
  ],
  "bypass_actors": [
    { "actor_id": <RECORDER_APP_ID>, "actor_type": "Integration", "bypass_mode": "always" }
  ]
}

Reconstruction from scratch

  1. Make sure the lean-eval-recorder app exists, is installed on the repo, and you know its App ID.
  2. Save the payload above to /tmp/main-ruleset.json, substituting the App ID.
  3. Apply: 
    gh api -X POST /repos/leanprover/lean-eval-submissions/rulesets \
    --input /tmp/main-ruleset.json

Staging Results ruleset

Create staging-results from the reviewed main commit before the first staging intake run. Ruleset Protect staging Results targets only refs/heads/staging-results, forbids deletion and non-fast-forward updates, requires ordinary writers to use a pull request, and gives the same lean-eval-recorder integration the sole always-bypass. It deliberately has no required status check: recorder commits contain staging data, not reviewed code, and never deploy the public leaderboard. The live ruleset ID created on 2026-08-23 is 21220656; reconstruct by copying the main protection payload, changing the name and include condition, and omitting required_status_checks.

Acceptable consequences of the bypass

  • The record job's results push to main does not drive the leaderboard implicitly: that redeploy is driven by an explicit results-advanced repository_dispatch. A Results-only push does select unfiltered ci.yml and the reviewer-gated, deployment-free immutable-tag promoter so the final accepted-result commit can become an exact historical-inventory cutoff; it does not select the Worker deployment workflow.
  • Anyone who can land a PR that modifies submission.yml to push arbitrary content to main could, after merge, exfiltrate that capability. This is the same trust boundary as merging any PR.