Repository navigation
Restore scheduled data refresh after an Arena retirement and feed rotation - #681
Merged
Merged
Conversation
The scheduled AI Charts data refresh has been unhealthy since 2026-10-06 for two reasons: - Arena retired "grok-imagine-image-2.0 (low)" from its text_to_image and image_edit leaderboards. The retention guard rejected any lost configuration. A newer publication may now retire a bounded few (10%, at least two), which are logged. A loss within the same publication or a larger one still stops for review. - Validation tests pinned specific models to the homepage activity feed, which rotates on every refresh. They now check the feed's shape and that each model links to a published route. This also commits the refreshed snapshots (AA Intelligence v4.3.2 at 113/108 configurations, DeepSWE, Terminal-Bench, Vals, Arena), plus the regenerated model route allowlist and social kit, which the refresh workflow cannot write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0thernet
enabled auto-merge (squash)
October 9, 2026 16:05
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The scheduled AI Charts data refresh has reported unhealthy since 2026-10-06 (issue #404). Two steps failed.
Root causes
atlas:arena-media:refresh):Arena text_to_image lost a model configuration. Upstream LMArena retiredgrok-imagine-image-2.0 (low)fromtext_to_image(2026-10-07 publication) andimage_edit(2026-10-06).hasArenaMediaChangedrejected any lost configuration, so every run failed. A newer publication may now retire up to max(2, 10%) configurations, and each one is logged. A loss within the same publication, or a larger one, still fails closed as a likely partial read.bun run test):app/home.test.tsxexpected/models/xiaomi/mimo-v2-6-pro/index, andlib/home-activity-feed.test.tsexpected Ling 3.1 Flash and Claude Opus 5.5, in the "Recent models" feed. That feed is the 6 newest models, so it rotates whenever the data changes. Both tests now check that every feed link renders and that each model links to aPUBLIC_MODEL_CARD_PATHSroute. The Opus 5.5 →claude-opus-5/negative check is kept.Data refresh included
The refresh workflow may only write
data/*.json. Today's AA Intelligence v4.3.2 roster change (113 / 108 positive-cost; Claude Haiku 5.5 and Mistral Large 4 added) also requires the derivedPUBLIC_MODEL_CARD_PATHSandkb/launch/social-kit.md(bun run launch:kit), as in #636. Those files and the refreshed snapshots are committed here: Arena, AA Intelligence v4.3.2, DeepSWE, Terminal-Bench, and Vals. The local calculator refresh failed on a Vast.ai offer count, so calculator inputs are unchanged.Verification
check:generated,check:cost-surfaces,typecheck,lint: green.bun run test: 2506/2507. The one failure waslib/gpt-6-1-sol-placement.property.test.ts, a fast-check counterexample with a denormalaaIndex(7.4e-323). It is unrelated to this change, and three reruns passed.🤖 Generated with Claude Code