scout design

[rosikand]

Making an ai agent/scout to help keep track of the flood of pr's and leaderboard updates.

Design

scout/                                                                                                                                                          
  state/        
    leaderboard.json        # ranked entries, filterable
    techniques.json         # technique registry with effectiveness data                                                                                        
    history.json            # daily snapshots
    explored.json           # what we've tried, what worked/didn't                                                                                              
  briefings/                                                                                                                                                    
    2026-03-23.md           # daily briefing, readable as-is
    2026-03-24.md                                                                                                                                               
  raw/                                                                                                                                                          
    prs/                    # cached raw PR data (JSON per PR)                                                                                                  
    issues/ 

[rosikand]

for the pr fetching, should we also include the new code files/changed code from the pr so we have that state as well while we iterate?

⏺ Good instinct, but I'd recommend not doing it in the daily scout — and here's why:

1. The PR bodies are already very detailed for this competition — they include hyperparameters, technique descriptions, ablation tables, architecture configs. That's what we need for scouting.
2. Volume problem — many PRs modify train_gpt.py which is the entire training script. 484 diffs would be massive and expensive to analyze with Claude.
3. Most diffs are variants of the same file — everyone forks the same train_gpt.py and tweaks it. The interesting info is what they changed (in the body), not the raw diff.

Better approach: fetch diffs on-demand when we decide to go deep on a specific PR. Once the scout identifies "PR openai#473's technique looks promising, let's try it," we pull that specific diff and study the
implementation. I can add a simple helper for that:

python scout.py --diff 473 # fetch and save the diff for a specific PR

This keeps the daily scout fast/cheap, and we go deep only when it matters. Want me to add that command?