Help us build top 10 LLM in the world while keeping it fully open source, which will accelerate everyone and everything that uses LLMs (science, technology, medicine, startups, businesses, etc.)
Check out our contributors leaderboard!
Our goals:
- GPT-1 Level by Dec 20 2025 ✓ Watch
- GPT-2 Level by Jan 20 2026
- GPT-3 Level by Apr 20 2026
- Top 150 in LMArena (GPT-4o-mini level) by June 2026
- Top 50 by Apr 2027
- Top 10 by Dec 2027
- We could aim for Top 1 by 2028, TBD
Can you make our LLM train faster and better?
👉 Full Setup Guide | Leaderboard
- Once you measure an improvement over the baseline according to the Setup Guide, submit your code in a GitHub pull request.
- The LLM must train faster or achieve lower loss on any of the benchmarks (8M, 20M, 100M, 1B tokens).
- Lower loss takes priority over training speed because pretraining data is limited - if your submission trains slower but achieves better (lower) loss for the same amount of tokens, it will probably be accepted, and vice versa.
- Add as little code as possible, keep it clean, rewrite AI generated pull request descriptions to increase quality.
- Submissions are judged case by case, tradeoffs between speed / loss etc. will be taken into account.
If you want to write a research paper improving this project, or if you or someone you know has extensive research experience and wants to contribute to this open-source initiative, contact me.
We will partner with compute providers while keeping all research/engineering/code fully open source.
Potential partners include: Hugging Face, NVIDIA, Microsoft, Google, Amazon, Meta, IBM, Oracle, Alibaba, Tencent, Huawei, Baidu, CoreWeave, Lambda Labs, Hyperbolic, Stability AI, OpenAI, Anthropic, xAI, Cohere, Mistral AI, Graphcore, Tenstorrent, Intel, AMD, Dell Technologies, ai2, a16z, Sequoia Capital, and more.