Repository navigation
Description
Tranche: #2504 · Agenda: #1960 · Lane: H · Hardening · Task: H5 · Ceiling: 3 h (a limit, not a target) · Tier: expected T1; node tools/proof/tier.mjs decides · Depends on: #2524 (V6) merged (for the viewer count)
Why this tranche exists (keep it in view): The system must scale smoothly across Souls, agents and viewers, each without starving the others and without limiting anyone's experience, far enough to be confident on a front-page day or at a live event with hundreds of active agents. This task is one step toward that. If it stops serving that, hand back. Overview:
docs/architecture/scaling.md; spec: #2504.
Main purpose
Make sure that during a spike a person is told, by an alert that leaves the machine, about viewers, the launcher, sign-ups and disk.
Problem
There is no metric for viewers per world, for the launcher's own event-loop delay, for worlds created, or for free disk. No unit in tools/world-launcher/deploy/ runs telemetry.mjs collect --alerts. telemetry/alerts.mjs says "No delivery: a local event only", and there is no dead-man check, so an alert that fires reaches nobody.
Outcome
Four new series, four alert rules, a unit that runs the collector, and a firing alert raises a host problem code so that hostOk is false on the public health endpoint, which the outside check (the tranche's hands-on issue) watches.
Design (given)
tools/world-launcher/metrics.mjs:sim_world_viewers{world}(the launcher'slivemap),sim_launcher_event_loop_delay_p99_seconds,sim_site_worlds_created_total,sim_host_disk_free_bytes.- Rules in the telemetry rules table: launcher loop delay over 250 ms; more than 10 worlds created in an hour; disk free under 20 %; viewers on one world over 1,000 (recorded in
alerts.jsonlonly, since there is no "information" level; it does not raise the code). - On a rule firing (except the viewers one), write a host problem code (
alert:<rule>) through the same file the health check uses, cleared when the rule resolves. sim-worlds-telemetry@.serviceindeploy/, installed byinstall.shand not enabled (enabling is hands-on).
Alternatives you may choose
- Deliver from the collector itself to a webhook. Rejected for now: a monitor on the same machine cannot report that machine failing; the outside check can.
Risks and footguns
install.shruns on every release, so a changed unit file goes live for every stage: keep "installed, not enabled".- Labels: world slug only. No Soul ids, account ids or addresses.
Acceptance
- A1 Tests for each series and each rule (fire and resolve).
- A2 Test: a firing rule makes
/__sim/healthzreporthostOk: falsewith the code, and resolving clears it. - A3
docs/operations/telemetry.mddocuments the series, rules, the unit and how to enable it.
Measure
Exempt from a performance bar. Evaluation cost was 40 ms a scrape with 400 worlds; report it again with the new rules.
Not in this task
The outside check itself. Enabling the unit.
Touches
tools/world-launcher/metrics.mjs, telemetry.mjs, telemetry/, launcher.mjs, deploy/ (the unit, install.sh), deploy/host-health.mjs, tests, docs/operations/telemetry.md, changes/.
Proof
By tier.mjs. The named tests.
Latitude and hand-back (tranche rule, the maintainer, 16 Nov)
- The main purpose above is the task. At every hour of work, and before any sub-step longer than 20 minutes, write one line in your notes: what you are doing and how it serves the main purpose. If you cannot say, stop that sub-step.
- Step 0 first, where the task has one. If the premise's number does not reproduce within a factor of two, hand back.
- You may change how: names, data shapes, test structure, an internal step, one of the listed alternatives, or your own equivalent, when the code as found shows it is better. Record it in the pull request under "Deviations from the design" with the evidence. Add your own analysis where it helps.
- You may not change what: the purpose, the acceptance bars, Touches or the ceiling. Never tune toward the expected effect; a result far from it is a finding.
- Even and uneven. Where this task compares runs, run the even arm it names and one uneven arm (the
eventshape of the shapes task, once merged; for viewers, arrivals in a wave with viewers who leave), and report both. Real load is skewed, spiky and shifting; a gain that shows only under even load is reported as that. - A smaller gain is still a result. If the measured improvement is real (outside the noise of two repeats) but below a bar, mark that box
unmet:with the numbers and finish the pull request. The coordinator weighs the gain against the complexity added and decides whether it merges (the maintainer, 16 Nov: any tangible improvement counts unless its complexity outweighs it). - Hand back at once (comment here, push what exists, end the session) when: Step 0 fails; the design does not fit the code and no alternative does; the expected effect will clearly not appear; you pass one and a half times your own estimate with acceptance not in sight; or you find the purpose itself is wrong. Say what you found, what you tried, the numbers, and one proposed rescope. A hand-back is a complete and valid outcome.
Stop rule
Say when you started in your first comment here, and state the time you took in the pull request. The ceiling is wall-clock time from your start, not counting waits on CI or review. Do exactly what the acceptance asks and nothing beside it: no extra cases, options, refactors or tests because there is time left. Stop when any of these happens: every acceptance box is ticked, marked unmet: or marked deferred:, and the pull request is ready; the ceiling is reached; the change needs more than a mechanical edit of ten lines in a file outside Touches (an "Also fixed" fix in the same area may take up to about 50), or a criterion not listed here; a declared run fails twice. Then push, comment here with what is done and the exact next step, and end the session. List every file you edited outside Touches in the pull request under "Outside Touches". Do not take another issue or re-declare a run. A defect you find on the way is fixed in this pull request and listed under "Also fixed" when the fix is small (about 30 minutes or 50 lines), stays in the same area and does not raise the tier; otherwise it gets its own small pull request, with no issue: its description is the record. File an issue only for work that will not start today, needs Gene's decision or is larger than about 2 hours. Only the coordinator changes Acceptance, Touches or the ceiling, by editing this issue.
Activity
genekogan commentedon Nov 16, 2026
Lane started 07:57 UTC 16 Nov (ceiling 3 h). Plan: the four series in metrics.mjs, four rules in the telemetry rules table, a firing rule (not the viewers one) writes
alert:<rule>through host-health's file sohostOkgoes false, andsim-worlds-telemetry@.serviceinstalled, not enabled. The coordinator added one item inside the purpose: a failed hourly backup (from the hourly-backup PR) shows in health and alerts like the nightly one. V6 is not merged yet: the viewer series reads the launcher'slivemap as it is on main. Exempt from a performance bar; I will report the scrape cost with the new rules from a focused test.genekogan commentedon Nov 16, 2026
PR #2584 (T1, ready, ci:run). A1–A3 met. Also in it: the coordinator's addition, a failed hourly backup raising
backup-hourly:<stage>and showing on the health page. Deviations are recorded in the PR: a disk size series beside disk free,refusalsstays information-only likeviewers-high, and the code travels in<data>/alert-problems.json. Evaluation cost with the new rules: about 41 ms median against 39 ms before (400 worlds). Gene's steps stay hands-on: enablesim-worlds-telemetry@production(the commands are in docs/operations/telemetry.md "The unit") and set up the outside check onhostOk. Next step: CI, then merge:ready.