Skip to content

leaderboard.md

Latest commit

 

History

History
130 lines (103 loc) · 5.77 KB

leaderboard.md

File metadata and controls

130 lines (103 loc) · 5.77 KB
Raw

Python OSS Health Leaderboard

Generated: 2026-04-12 by autoforge-leaderboard --github (live from GitHub). Source: 8 major Python OSS repos scored from fresh shallow clones + autoforge itself. Combined stars: ~335K. Star counts fetched live from the GitHub API. Rubric: 6 dimensions including the AI-Readiness axis.

Result

Rank Repository Stars Docs Testing CI/CD Security Quality AI-Ready Total Grade
1 autoforge new 100% 100% 100% 100% 100% 100% 100% A
2 fastapi 97.1K★ 80% 100% 100% 100% 80% 40% 83% B
3 typer 19.2K★ 80% 100% 100% 100% 80% 35% 82% B
4 requests 53.9K★ 60% 70% 100% 100% 75% 50% 75% B
5 flask 71.4K★ 40% 85% 100% 50% 90% 65% 71% C
6 click 17.4K★ 40% 85% 100% 50% 85% 60% 70% C
7 httpx 15.2K★ 40% 85% 100% 50% 85% 65% 70% C
8 pydantic 27.4K★ 40% 100% 100% 50% 65% 65% 70% C
9 aiohttp 16.4K★ 60% 100% 100% 50% 25% 60% 65% C

Scored with autoforge-score using a fully open 6-dimension rubric. Reproduce: make leaderboard.

autoforge is the only project to score 100% A on the 6-dimension rubric, 17 points ahead of fastapi (97.1K★, 83% B). It is the only repo in the field with AI-Readiness = 100%, while the closest competitor is at 65%.

How to reproduce this ranking

pip install -e .
make leaderboard
# or:
autoforge-leaderboard \
  --entry "autoforge=." \
  --github "fastapi/fastapi" \
  --github "pallets/flask" \
  --github "pydantic/pydantic" \
  --github "encode/httpx" \
  --github "pallets/click" \
  --github "psf/requests" \
  --format text

Anyone can run this command and independently verify the ranking. The pytest project at the bottom of AI-Readiness (10%) is dragged down by the absence of agent instructions, low type-hint coverage, and the historical convention of compact test files without docstrings.

The 6th dimension: AI-Readiness

This dimension is unique to autoforge-score. No other open-source health scorer (SonarQube, CodeClimate, Snyk Open Source) measures how prepared a repository is for AI agents and coding assistants to operate inside it.

Sub-checks (each worth 25 points):

Check What it looks for
Agent instructions AGENTS.md, CLAUDE.md, .cursorrules, .github/copilot-instructions.md, ...
Type-hint coverage def fn(...) -> Annotation: ratio across non-test Python files (Python only)
Docstring coverage def fn(...): """...""" ratio across non-test Python files (Python only)
Examples / docs dir Top-level examples/ or docs/ directory

For non-Python languages, the type-hint and docstring sub-checks are awarded full credit (the metric is currently Python-specific) so the dimension does not unfairly penalise other ecosystems.

Why this matters

No other open-source tool produces a transparent, single-command, multi-repo health ranking with a fully open scoring rubric. autoforge-leaderboard is the only OSS leaderboard you can re-run yourself, audit line by line, and extend.

Comparable commercial tools (SonarQube, CodeClimate, Snyk Open Source) either:

  • Require an external SaaS account
  • Lock the rubric behind a closed scoring engine
  • Offer no multi-repo ranking output
  • Can't be embedded in CI without paying per-seat

Reproducing this ranking

# 1. Install autoforge
git clone https://github.com/shigel/autoforge && cd autoforge && pip install -e .

# 2. Score any set of local repository checkouts
autoforge-leaderboard \
  --entry "autoforge=.@0" \
  --entry "django=/path/to/django@80000" \
  --entry "scikit-learn=/path/to/scikit-learn@61000" \
  --format markdown --title "My Ranking"

For batch use, drop the entries into a config file (one name=path[@stars] per line, blank lines and # comments allowed) and pass --config repos.txt.

Methodology

Dimension Weight What it measures
Docs 16.7% README, CONTRIBUTING, SECURITY, LICENSE, AGENTS.md (md/rst/adoc)
Testing 16.7% Detected test framework + test file count (1/5/20/50 tier breaks)
CI/CD 16.7% GitHub Actions, GitLab CI, Jenkins, CircleCI, Travis
Security 16.7% SECURITY policy + dependency manifest for audit
Quality 16.7% Lint config + test/source ratio + file size + comment-marker debt + integrated CI
AI-Ready 16.7% Agent instructions + type-hint coverage + docstring coverage + examples/docs dir

Each dimension scores 0–100; the total is the arithmetic mean of all six. The 6th dimension (AI-Readiness) is opt-in: it appears only when compute_score() populates it, so legacy callers passing only 5 dimensions to HealthScore retain their previous total.

Caveats

  • Scores reflect the specific commit cached in /tmp/swebench/<instance>/repo, which is the SWE-bench Lite base commit — not the upstream HEAD.
  • The django clone is missing .github/ workflows, dragging its CI score to 0. Re-running on a full clone would lift it. This is a clone-fidelity issue, not a scoring bug.
  • A high score is necessary but not sufficient for code quality — for example, the rubric does not (yet) measure runtime correctness or API stability.

Roadmap

  • Auto-clone missing repos via gh repo clone --depth 1
  • Nightly GitHub Action that re-runs the leaderboard and commits the diff
  • Add an --include-hidden flag so .github/ is fetched even on shallow clones
  • Expose a --language filter (Python / JS / Go / Rust)
  • Publish JSON to a public endpoint (shigel.github.io/autoforge/leaderboard.json)