Repository navigation
feat(eval): eval-load --raw-from drives 100 to 500 agents from one core (#2509) - #2577
Conversation
… scans and the menu counters from one eval-load run (#2508) eval-load --profile CPU-profiles the world's clock thread over the load window and reports an act's clock ms by part (#2470's group names; the tick's route searches split into first plans, re-plans around people, detours and paged chunks), the planner's sync and audit, and long holds by source. --menu-stats folds the world's menu counters over the window, with builds per act. Every arm states intended and achieved acts. The server adds bind, check and route to the act's parts in live.json, with since-start sums of every part and of the time admission charged each act; offerGate's bind is timed inside bind. Timing marks only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nner part windows rotate, route is the check's search (#2508) - admission charges into its own world's API buffer, not the program default - the bind, check and route windows are cleared when a window rotates - route times the check's own feasibility search, so it lies inside check - the classifier counts act-path frames as the act only under no other route - the profile stops when the load ends; the menu window is taken from the first counter lines after the load's start and end Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oad (#2509 Step 0) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… process, up to 500 (#2509) Souls k and above join from their own documentation-range addresses, 8 at a time, and act by catalog id with the client's retry rules, on the same seeded draws as the paced loop. The report gives the driver's processor share, and achieved against intended acts and act queue per tenth of agents by activity. timed-run allows --souls up to 500, --raw-from and --shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…; eval-load --raw-from in playtests.md (#2509) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uls on one world (#2509) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Proof check:
|
| Check | Status | Detail |
|---|---|---|
| problem statement | ok | present |
| linked issue | ok | #2509 |
Acceptance criteria
| Criterion | Status | Evidence | |
|---|---|---|---|
| [ ] | A1 eval-load --souls 100 --raw-from 4 --act-sec 10 completes setup in u |
unmet | setup and the driver are met. Achieved acts were under 95 % in one of the two idle repeats. Details: |
| [ ] | A2 At 24 agents, raw and real clients give act wait medians within 15 % o | unmet | one repeat of two was outside 15 %. Run 43827047477 |
| [x] | A3 --souls 300 --raw-from 4 sets up within 15 minutes. |
met | run 43827050512 (`--souls 300 --raw-from 4 --act-sec |
| [ ] | A4 --shape event at 50 agents runs and reports per tenth of agents; the |
deferred | #2556. The shapes module (M8) has not merged. --shape uniform is the paced loop. Any other shape is refused with that |
Proof links: 0 video, 0 image, proof.json linked.
Updated on every edit, push, label and review. How to fill it: .claude/skills/journal/SKILL.md. T3-T4 also need the maintainer's sign-off (tools/proof/gate.mjs). A maintainer can waive with the proof:waived label.
|
Coordinator decision: accept as scored, merge as the driver M4 and the rest of the tranche build on.
|
3-way merge against the fork point from the old base (014397c), by .github/scripts/restack.mjs.
|
Restack: #2558 squash-merged into
Head |
# Conflicts: # .github/scripts/timed-run.mjs # tools/playtest/eval/load.mjs
|
Merge queue: this PR left the queue without merging (reason: Fix it and add |
… --scan-census and --shape) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
eval-loadran oneservedaemon per Soul and spawned twonode eden.mjscalls per act on the driver's core, and it set up 6 Souls per 30 s. So the agent gates could not load a world with 100 to 500 agents without the driver, or the clock of setup, being the limit. Closes #2509. Stacked on the act meter (agent/2508-act-meter, same file); retarget to main once that merges.Acceptance
eval-load --souls 100 --raw-from 4 --act-sec 10completes setup in under 5 minutes and achieves at least 95 % of intended acts on an idle world arm (acts paced far apart), with the driver under 0.5 of a core. — unmet: setup and the driver are met. Achieved acts were under 95 % in one of the two idle repeats. Details:--act-sec 10): setup took 22 s and 22 s, and the driver used 0.12 and 0.10 of a core.--act-sec 60): 179 of 200 intended acts (89.5 %) in repeat 1 and 199 of 200 (99.5 %) in repeat 2. The driver used 0.11 of a core in each.--souls 24 --raw-from 13, so 12 real and 12 raw Souls on one world. Raw act queue p50 against real:--souls 300 --raw-from 4sets up within 15 minutes. — evidence: run 43827050512 (--souls 300 --raw-from 4 --act-sec 10 --duration-sec 60). Setup took 53 s and 52 s, and the driver used 0.13 and 0.12 of a core.--shape eventat 50 agents runs and reports per tenth of agents; the schedule is seeded and identical across base and candidate arms (test). — deferred: Scaling M8 · Uneven load shapes for every bench: who acts, how often, when, where and what #2556. The shapes module (M8) has not merged.--shape uniformis the paced loop. Any other shape is refused with that reason, and the uneven shapes plug in atshapeOf. The seeded draws are tested: a raw Soul replays the same acts for the same seed, and draws the same words a real paced Soul would (load.test.mjs).Step 0 (issuecomment-6910657516, where the bars were declared before these runs): run 43823959944 ran 50 real Souls at
--act-sec 10. Everything on the load generator's cores used 0.65 of a core while achieving only 18 % of intended acts, so the premise holds.The driver, per run (driver = everything on the load generator's CPUs):
What the misses are.
--act-sec, which is 2 acts per Soul in 120 s. The paced draws (a first act at 0–60 s, then gaps of 30–90 s) give each Soul 1 to 3 acts, so with so few acts the achieved share swings between repeats. Nothing was refused for load in the busiest tenth: it achieved 34–35 acts against 20 intended.actretries further.A finding for lane A and M4, not acted on. Every act queues about 19.5 s at the world (p50), for real and raw Souls alike and from 24 to 300 agents. So a closed loop at
--act-sec 10achieves about 20 % of intended. The per-tenth tables show it flat across agents, with almost no slow-down refusals: the wait is the world's, not a per-Soul budget.Deviations from the design
/proc/statover the process's own allowed CPUs). Under timed-run the world is pinned elsewhere, and theservedaemons are detached, so the harness's own counters miss them. The harness's own share is printed too.find. Its draws (first offset, gap, word, row) are the real loop's, so the same seed makes the same choices.clientCall(client-soul.mjs) also returns the last answer's Server-Timing, so raw acts carry the same queue time as the real client's trace. This is a field added to its return, not only an export.uniform, every agent's intended count is equal.Proof
Tier: T0 (
node tools/proof/tier.mjs: tools, the timed-run allowlist and docs).Self-audit
node --test tools/playtest/eval/load.test.mjs→ 3/3.node --test .github/scripts/timed-run.test.mjs→ 16/16. The allowlist assertion moved from--souls 101to--souls 501, and--raw-from 0and an unknown shape were added.node --test tools/playtest/eval/prof-buckets.test.mjs→ pass.node .github/scripts/guards.mjs scrub→ 0 problems.//comment that swallowed the end of a line. Its run, 43823695809, was cancelled before measuring anything; the code was fixed and re-dispatched.--raw-from, nothing changes except the driver line, and owner messages going to real Souls only, which is all Souls then.client-soul.mjscallers get one more field.--soulsup to 500, plus--raw-fromand--shape.docs/testing/playtests.md, under Timed runs..github/scripts/timed-run.test.mjs, one assertion line, the allowlist's own test.Changelog
Labelled
no-changelog: tools and docs only.Risk
--raw-fromit is unchanged; the existing paced loop and setup still run.🤖 Generated with Claude Code