Skip to content

feat(eval): eval-load --raw-from drives 100 to 500 agents from one core (#2509) - #2577

Merged
genekogan merged 10 commits into
mainfrom
agent/2509-raw-driver
Nov 16, 2026
Merged

genekogan merged 10 commits into
mainfrom
agent/2509-raw-driver

Conversation

@genekogan

@genekogan genekogan commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Problem

eval-load ran one serve daemon per Soul and spawned two node eden.mjs calls per act on the driver's core, and it set up 6 Souls per 30 s. So the agent gates could not load a world with 100 to 500 agents without the driver, or the clock of setup, being the limit. Closes #2509. Stacked on the act meter (agent/2508-act-meter, same file); retarget to main once that merges.

Acceptance

  • A1 eval-load --souls 100 --raw-from 4 --act-sec 10 completes setup in under 5 minutes and achieves at least 95 % of intended acts on an idle world arm (acts paced far apart), with the driver under 0.5 of a core. — unmet: setup and the driver are met. Achieved acts were under 95 % in one of the two idle repeats. Details:
    • Busy arm (run 43827039030, --act-sec 10): setup took 22 s and 22 s, and the driver used 0.12 and 0.10 of a core.
    • Idle arm (run 43827044255, --act-sec 60): 179 of 200 intended acts (89.5 %) in repeat 1 and 199 of 200 (99.5 %) in repeat 2. The driver used 0.11 of a core in each.
  • A2 At 24 agents, raw and real clients give act wait medians within 15 % of each other on the same world (one run with half of each). — unmet: one repeat of two was outside 15 %. Run 43827047477 ran --souls 24 --raw-from 13, so 12 real and 12 raw Souls on one world. Raw act queue p50 against real:
    • repeat 1: 15194 ms against 19298 ms (−21 %);
    • repeat 2: 9901 ms against 10155 ms (−3 %).
  • A3 --souls 300 --raw-from 4 sets up within 15 minutes. — evidence: run 43827050512 (--souls 300 --raw-from 4 --act-sec 10 --duration-sec 60). Setup took 53 s and 52 s, and the driver used 0.13 and 0.12 of a core.
  • A4 --shape event at 50 agents runs and reports per tenth of agents; the schedule is seeded and identical across base and candidate arms (test). — deferred: Scaling M8 · Uneven load shapes for every bench: who acts, how often, when, where and what #2556. The shapes module (M8) has not merged. --shape uniform is the paced loop. Any other shape is refused with that reason, and the uneven shapes plug in at shapeOf. The seeded draws are tested: a raw Soul replays the same acts for the same seed, and draws the same words a real paced Soul would (load.test.mjs).

Step 0 (issuecomment-6910657516, where the bars were declared before these runs): run 43823959944 ran 50 real Souls at --act-sec 10. Everything on the load generator's cores used 0.65 of a core while achieving only 18 % of intended acts, so the premise holds.

The driver, per run (driver = everything on the load generator's CPUs):

run Souls (raw) setup s driver cores acts achieved / intended act queue p50 ms
Step 0, real only 50 (0) about 270 (6 per 30 s) 0.65 107 / 600 19.5 k
A1 busy r1 / r2 100 (97) 22 / 22 0.12 / 0.10 277 / 1200, 232 / 1200 19.1 k / 19.4 k
A1 idle r1 / r2 100 (97) 23 / 23 0.11 / 0.11 179 / 200, 199 / 200 19.0 k / 19.0 k
A2 r1 / r2 24 (12) 38 / 36 0.27 / 0.43 83 / 288, 245 / 288 real 19.3 k vs raw 15.2 k; real 10.2 k vs raw 9.9 k
A3 r1 / r2 300 (297) 53 / 52 0.13 / 0.12 318 / 1800, 329 / 1800 19.5 k / 19.4 k

What the misses are.

  • A1's idle arm. The intended count is Souls × seconds ÷ --act-sec, which is 2 acts per Soul in 120 s. The paced draws (a first act at 0–60 s, then gaps of 30–90 s) give each Soul 1 to 3 acts, so with so few acts the achieved share swings between repeats. Nothing was refused for load in the busiest tenth: it achieved 34–35 acts against 20 intended.
  • A2's repeat 1.
    • Raw Souls achieved fewer acts (45 against 83).
    • A raw act the world refused busy three times ends without a queue time, while the real client's act retries further.
    • On the same world, repeat 2 agreed within 3 %.

A finding for lane A and M4, not acted on. Every act queues about 19.5 s at the world (p50), for real and raw Souls alike and from 24 to 300 agents. So a closed loop at --act-sec 10 achieves about 20 % of intended. The per-tenth tables show it flat across agents, with almost no slow-down refusals: the wait is the world's, not a per-Soul budget.

Deviations from the design

  • The driver is measured as everything on the load generator's CPUs (/proc/stat over the process's own allowed CPUs). Under timed-run the world is pinned elsewhere, and the serve daemons are detached, so the harness's own counters miss them. The harness's own share is printed too.
  • A raw Soul resolves a paced word from the catalog once: places, spots and seats by name. It does not use the CLI's nearest-first find. Its draws (first offset, gap, word, row) are the real loop's, so the same seed makes the same choices.
  • Raw Souls join with the dev identity flow over HTTP (create, wait out a pending arrival, pair), each from its own documentation-range address (192.0.2/24, 198.51.100/24, 203.0.113/24; at most 762).
  • clientCall (client-soul.mjs) also returns the last answer's Server-Timing, so raw acts carry the same queue time as the real client's trace. This is a field added to its return, not only an export.
  • Raw Souls get no owner messages, and the report says how else they differ.
  • The tenths split agents evenly by intended, then achieved. Under uniform, every agent's intended count is equal.

Proof

Tier: T0 (node tools/proof/tier.mjs: tools, the timed-run allowlist and docs).

Self-audit

  • Checks run:
    • node --test tools/playtest/eval/load.test.mjs → 3/3.
    • node --test .github/scripts/timed-run.test.mjs → 16/16. The allowlist assertion moved from --souls 101 to --souls 501, and --raw-from 0 and an unknown shape were added.
    • node --test tools/playtest/eval/prof-buckets.test.mjs → pass.
    • node .github/scripts/guards.mjs scrub → 0 problems.
    • Not run locally: any load run. All runs were on souls-timed.
  • Failed attempts: the first Step 0 commit had a syntax error, a // comment that swallowed the end of a line. Its run, 43823695809, was cancelled before measuring anything; the code was fixed and re-dispatched.
  • Long tail / externalities:
    • Without --raw-from, nothing changes except the driver line, and owner messages going to real Souls only, which is all Souls then.
    • client-soul.mjs callers get one more field.
    • timed-run allows --souls up to 500, plus --raw-from and --shape.
  • Known limits:
    • A1 and A2 each missed in one repeat (above).
    • There is no uneven arm until the shapes module merges.
    • The 19.5 s act queue limits what any closed-loop run achieves.
  • Docs: docs/testing/playtests.md, under Timed runs.
  • Time: started 07:22 UTC (issue comment). Work ran 07:22–07:54 and 08:47–08:52. I waited about 55 min on runners in between. That is about 40 min of work and 1 h 30 min of wall clock, against a 3 h ceiling.
  • Outside Touches: .github/scripts/timed-run.test.mjs, one assertion line, the allowlist's own test.

Changelog

Labelled no-changelog: tools and docs only.

Risk

  • What could break. A raw Soul could behave unlike the real client and skew a gate. The differences are stated in every report, and A2 measures them on one world.
  • The real path. Without --raw-from it is unchanged; the existing paced loop and setup still run.
  • Rollback. Revert the PR.

🤖 Generated with Claude Code

genekogan and others added 7 commits November 16, 2026 01:31
… scans and the menu counters from one eval-load run (#2508)

eval-load --profile CPU-profiles the world's clock thread over the load window
and reports an act's clock ms by part (#2470's group names; the tick's route
searches split into first plans, re-plans around people, detours and paged
chunks), the planner's sync and audit, and long holds by source. --menu-stats
folds the world's menu counters over the window, with builds per act. Every arm
states intended and achieved acts.

The server adds bind, check and route to the act's parts in live.json, with
since-start sums of every part and of the time admission charged each act;
offerGate's bind is timed inside bind. Timing marks only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nner part windows rotate, route is the check's search (#2508)

- admission charges into its own world's API buffer, not the program default
- the bind, check and route windows are cleared when a window rotates
- route times the check's own feasibility search, so it lies inside check
- the classifier counts act-path frames as the act only under no other route
- the profile stops when the load ends; the menu window is taken from the
  first counter lines after the load's start and end

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oad (#2509 Step 0)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… process, up to 500 (#2509)

Souls k and above join from their own documentation-range addresses, 8 at a
time, and act by catalog id with the client's retry rules, on the same seeded
draws as the paced loop. The report gives the driver's processor share, and
achieved against intended acts and act queue per tenth of agents by activity.
timed-run allows --souls up to 500, --raw-from and --shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…; eval-load --raw-from in playtests.md (#2509)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uls on one world (#2509)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@genekogan genekogan added the no-changelog Nothing user-facing; no changes/ fragment needed label Nov 16, 2026
@genekogan
genekogan marked this pull request as ready for review November 16, 2026 08:50
@genekogan genekogan added the ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Proof check: proof-ok passes

Tier T0 from .github/scripts/timed-run.mjs, .github/scripts/timed-run.test.mjs, docs/testing/playtests.md, tools/bench/capacity/meetup/client-soul.mjs, ... (node tools/proof/tier.mjs).
Required at T0 (docs/process/PROOF-POLICY.yml): lint.

Check Status Detail
problem statement ok present
linked issue ok #2509
Acceptance criteria
Criterion Status Evidence
[ ] A1 eval-load --souls 100 --raw-from 4 --act-sec 10 completes setup in u unmet setup and the driver are met. Achieved acts were under 95 % in one of the two idle repeats. Details:
[ ] A2 At 24 agents, raw and real clients give act wait medians within 15 % o unmet one repeat of two was outside 15 %. Run 43827047477
[x] A3 --souls 300 --raw-from 4 sets up within 15 minutes. met run 43827050512 (`--souls 300 --raw-from 4 --act-sec
[ ] A4 --shape event at 50 agents runs and reports per tenth of agents; the deferred #2556. The shapes module (M8) has not merged. --shape uniform is the paced loop. Any other shape is refused with that

Proof links: 0 video, 0 image, proof.json linked.

Updated on every edit, push, label and review. How to fill it: .claude/skills/journal/SKILL.md. T3-T4 also need the maintainer's sign-off (tools/proof/gate.mjs). A maintainer can waive with the proof:waived label.

@genekogan

Copy link
Copy Markdown
Member Author

Coordinator decision: accept as scored, merge as the driver M4 and the rest of the tranche build on.

  • A3 met. A1 and A2 each missed in one of two repeats, and both misses trace to the world, not the driver:
    • A1's idle arm swings with the paced draws (2 acts per Soul in 120 s), and nothing was refused for load.
    • A2's repeat-1 gap comes from raw acts refused busy three times, which end without a queue time.
  • The finding stands for M4 and lane A: every act queues about 19.5 s at the world from 24 to 300 agents. That matches the absorbing admission state V0 is fixing (Scaling V0 · A viewer regression since 14 Nov: 50 viewers stop acts, 200 blow the heap (bisect and fix) #2559).
  • Not re-scored. M4 re-runs the raw-against-real comparison once V0's fix is on main, and records it there.

Base automatically changed from agent/2508-act-meter to main November 16, 2026 09:02
@genekogan genekogan added ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) and removed ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) labels Nov 16, 2026
3-way merge against the fork point from the old base (014397c), by .github/scripts/restack.mjs.
@genekogan

Copy link
Copy Markdown
Member Author

Restack: #2558 squash-merged into main.

main was merged forward with a 3-way merge against this branch's fork point from the old base (014397c), so files this PR does not change take the squashed content and the diff is this PR's own changes again.

Head 1fc6c65 → 85ae652 (fast-forward push, no force).

@genekogan genekogan added merge:ready Hand the PR to the merge train (.github/workflows/merge-train.yml) ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) and removed ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) labels Nov 16, 2026
@genekogan
genekogan enabled auto-merge November 16, 2026 09:45
@genekogan
genekogan added this pull request to the merge queue Nov 16, 2026
# Conflicts:
#	.github/scripts/timed-run.mjs
#	tools/playtest/eval/load.mjs
@genekogan genekogan added ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) and removed ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) labels Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 16, 2026
@genekogan genekogan removed the merge:ready Hand the PR to the merge train (.github/workflows/merge-train.yml) label Nov 16, 2026
@genekogan

Copy link
Copy Markdown
Member Author

Merge queue: this PR left the queue without merging (reason: MERGE_CONFLICT), so it left the train. The failing run is on the gh-readonly-queue commit in the Actions tab.

Fix it and add merge:ready again.

@genekogan genekogan added merge:ready Hand the PR to the merge train (.github/workflows/merge-train.yml) ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) and removed ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) labels Nov 16, 2026
@genekogan
genekogan enabled auto-merge November 16, 2026 11:59
@genekogan
genekogan added this pull request to the merge queue Nov 16, 2026
@genekogan genekogan added ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) and removed ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) labels Nov 16, 2026
Merged via the queue into main with commit e6e48fc Nov 16, 2026
62 of 68 checks passed
@genekogan
genekogan deleted the agent/2509-raw-driver branch November 16, 2026 13:02
genekogan added a commit that referenced this pull request Nov 16, 2026
… --scan-census and --shape)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:run Open the CI window: run this ready PR's heavy CI jobs (docs/testing.md#ci-window) merge:ready Hand the PR to the merge train (.github/workflows/merge-train.yml) no-changelog Nothing user-facing; no changes/ fragment needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scaling M3 · A raw-HTTP agent driver, so 100 to 500 agents can be driven from one core in even and uneven load

1 participant