Skip to content

Scaling H5 · The numbers a spike needs are exported, alerted on, and reach a person #2541

Closed

Description

@genekogan

Tranche: #2504 · Agenda: #1960 · Lane: H · Hardening · Task: H5 · Ceiling: 3 h (a limit, not a target) · Tier: expected T1; node tools/proof/tier.mjs decides · Depends on: #2524 (V6) merged (for the viewer count)

Why this tranche exists (keep it in view): The system must scale smoothly across Souls, agents and viewers, each without starving the others and without limiting anyone's experience, far enough to be confident on a front-page day or at a live event with hundreds of active agents. This task is one step toward that. If it stops serving that, hand back. Overview: docs/architecture/scaling.md; spec: #2504.

Main purpose

Make sure that during a spike a person is told, by an alert that leaves the machine, about viewers, the launcher, sign-ups and disk.

Problem

There is no metric for viewers per world, for the launcher's own event-loop delay, for worlds created, or for free disk. No unit in tools/world-launcher/deploy/ runs telemetry.mjs collect --alerts. telemetry/alerts.mjs says "No delivery: a local event only", and there is no dead-man check, so an alert that fires reaches nobody.

Outcome

Four new series, four alert rules, a unit that runs the collector, and a firing alert raises a host problem code so that hostOk is false on the public health endpoint, which the outside check (the tranche's hands-on issue) watches.

Design (given)

  • tools/world-launcher/metrics.mjs: sim_world_viewers{world} (the launcher's live map), sim_launcher_event_loop_delay_p99_seconds, sim_site_worlds_created_total, sim_host_disk_free_bytes.
  • Rules in the telemetry rules table: launcher loop delay over 250 ms; more than 10 worlds created in an hour; disk free under 20 %; viewers on one world over 1,000 (recorded in alerts.jsonl only, since there is no "information" level; it does not raise the code).
  • On a rule firing (except the viewers one), write a host problem code (alert:<rule>) through the same file the health check uses, cleared when the rule resolves.
  • sim-worlds-telemetry@.service in deploy/, installed by install.sh and not enabled (enabling is hands-on).

Alternatives you may choose

  • Deliver from the collector itself to a webhook. Rejected for now: a monitor on the same machine cannot report that machine failing; the outside check can.

Risks and footguns

  • install.sh runs on every release, so a changed unit file goes live for every stage: keep "installed, not enabled".
  • Labels: world slug only. No Soul ids, account ids or addresses.

Acceptance

  • A1 Tests for each series and each rule (fire and resolve).
  • A2 Test: a firing rule makes /__sim/healthz report hostOk: false with the code, and resolving clears it.
  • A3 docs/operations/telemetry.md documents the series, rules, the unit and how to enable it.

Measure

Exempt from a performance bar. Evaluation cost was 40 ms a scrape with 400 worlds; report it again with the new rules.

Not in this task

The outside check itself. Enabling the unit.

Touches

tools/world-launcher/metrics.mjs, telemetry.mjs, telemetry/, launcher.mjs, deploy/ (the unit, install.sh), deploy/host-health.mjs, tests, docs/operations/telemetry.md, changes/.

Proof

By tier.mjs. The named tests.

Latitude and hand-back (tranche rule, the maintainer, 16 Nov)

  • The main purpose above is the task. At every hour of work, and before any sub-step longer than 20 minutes, write one line in your notes: what you are doing and how it serves the main purpose. If you cannot say, stop that sub-step.
  • Step 0 first, where the task has one. If the premise's number does not reproduce within a factor of two, hand back.
  • You may change how: names, data shapes, test structure, an internal step, one of the listed alternatives, or your own equivalent, when the code as found shows it is better. Record it in the pull request under "Deviations from the design" with the evidence. Add your own analysis where it helps.
  • You may not change what: the purpose, the acceptance bars, Touches or the ceiling. Never tune toward the expected effect; a result far from it is a finding.
  • Even and uneven. Where this task compares runs, run the even arm it names and one uneven arm (the event shape of the shapes task, once merged; for viewers, arrivals in a wave with viewers who leave), and report both. Real load is skewed, spiky and shifting; a gain that shows only under even load is reported as that.
  • A smaller gain is still a result. If the measured improvement is real (outside the noise of two repeats) but below a bar, mark that box unmet: with the numbers and finish the pull request. The coordinator weighs the gain against the complexity added and decides whether it merges (the maintainer, 16 Nov: any tangible improvement counts unless its complexity outweighs it).
  • Hand back at once (comment here, push what exists, end the session) when: Step 0 fails; the design does not fit the code and no alternative does; the expected effect will clearly not appear; you pass one and a half times your own estimate with acceptance not in sight; or you find the purpose itself is wrong. Say what you found, what you tried, the numbers, and one proposed rescope. A hand-back is a complete and valid outcome.

Stop rule

Say when you started in your first comment here, and state the time you took in the pull request. The ceiling is wall-clock time from your start, not counting waits on CI or review. Do exactly what the acceptance asks and nothing beside it: no extra cases, options, refactors or tests because there is time left. Stop when any of these happens: every acceptance box is ticked, marked unmet: or marked deferred:, and the pull request is ready; the ceiling is reached; the change needs more than a mechanical edit of ten lines in a file outside Touches (an "Also fixed" fix in the same area may take up to about 50), or a criterion not listed here; a declared run fails twice. Then push, comment here with what is done and the exact next step, and end the session. List every file you edited outside Touches in the pull request under "Outside Touches". Do not take another issue or re-declare a run. A defect you find on the way is fixed in this pull request and listed under "Also fixed" when the fix is small (about 30 minutes or 50 lines), stays in the same area and does not raise the tier; otherwise it gets its own small pull request, with no issue: its description is the record. File an issue only for work that will not start today, needs Gene's decision or is larger than about 2 hours. Only the coordinator changes Acceptance, Touches or the ceiling, by editing this issue.

Activity

  1. added this to the Scaling tranche milestone on Nov 16, 2026
  2. self-assigned this
    on Nov 16, 2026
  3. added
    agent:claimedAn agent or person has claimed this issue; others leave it alone
    on Nov 16, 2026
  4. genekogan commented on Nov 16, 2026

    @genekogan
    MemberAuthor

    Lane started 07:57 UTC 16 Nov (ceiling 3 h). Plan: the four series in metrics.mjs, four rules in the telemetry rules table, a firing rule (not the viewers one) writes alert:<rule> through host-health's file so hostOk goes false, and sim-worlds-telemetry@.service installed, not enabled. The coordinator added one item inside the purpose: a failed hourly backup (from the hourly-backup PR) shows in health and alerts like the nightly one. V6 is not merged yet: the viewer series reads the launcher's live map as it is on main. Exempt from a performance bar; I will report the scrape cost with the new rules from a focused test.

  5. genekogan commented on Nov 16, 2026

    @genekogan
    MemberAuthor

    PR #2584 (T1, ready, ci:run). A1–A3 met. Also in it: the coordinator's addition, a failed hourly backup raising backup-hourly:<stage> and showing on the health page. Deviations are recorded in the PR: a disk size series beside disk free, refusals stays information-only like viewers-high, and the code travels in <data>/alert-problems.json. Evaluation cost with the new rules: about 41 ms median against 39 ms before (400 worlds). Gene's steps stay hands-on: enable sim-worlds-telemetry@production (the commands are in docs/operations/telemetry.md "The unit") and set up the outside check on hostOk. Next step: CI, then merge:ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent:claimedAn agent or person has claimed this issue; others leave it aloneepic:scalabilityServer scalability / reliability epictype:taskCurrent task

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions