Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 50 additions & 2 deletions docs/operations/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ node tools/world-launcher/telemetry.mjs report --dir <launcher data>/telemetry -
node tools/world-launcher/telemetry.mjs close --dir <launcher data>/telemetry # roll up and prune now
```

On the host, the `sim-worlds-telemetry@<stage>` unit runs the collector with alerts and the problems file, once the maintainer enables it ([below](#the-unit)).

`--url` must be a loopback address. The launcher answers `/__sim/metrics` to direct loopback requests only. The
collector stops cleanly on SIGINT/SIGTERM, and on restart it continues the current hour, recording the restart as a
boundary.
Expand Down Expand Up @@ -48,6 +50,19 @@ boundary.
- Only files inside `--dir` are ever removed. A failed write is counted and the collector carries on; the game never
waits on telemetry.

### The series a spike needs

The launcher exports four series for a spike (#2541), next to the world series from `live.json`:

| Series | Type | What |
|---|---|---|
| `sim_world_viewers{world}` | gauge | viewer sockets open on the world at the launcher (its `live` map); 0 is a reading |
| `sim_launcher_event_loop_delay_p99_seconds` | gauge | the launcher process's own event-loop delay p99 since the previous scrape |
| `sim_site_worlds_created_total` | counter | worlds created on the site since the launcher started (sign-ups and operator creations) |
| `sim_host_disk_free_bytes`, `sim_host_disk_size_bytes` | gauge | free and total bytes of the disk the launcher data is on |

Labels are the world slug only: never a Soul id, account id or address.

Size, measured with a synthetic scrape of 400 registered worlds (20 running): about 7 KB per scrape before gzip. At
15 s that is about 43 MB per day before gzip, well inside the cap.

Expand Down Expand Up @@ -118,7 +133,7 @@ Each rule has three windows, which give it hysteresis:
- **Clear window:** the condition must stay clear for this long before the rule resolves. The default is 60 s.
- **Clear threshold** (optional): a firing rule only counts as clear once its value is back past this threshold.

A running world without a rule's series becomes `unknown` after the hold window, never healthy. A world that is not running is not judged, except for crash loops. Nothing is sent anywhere: delivery and dead-man alerting belong to #461, and a monitor on the same host cannot detect that host failing.
A running world without a rule's series becomes `unknown` after the hold window, never healthy. A world that is not running is not judged, except for crash loops. A firing rule reaches a person through the public health check ([below](#reaching-a-person)). Nothing is sent from the host itself: a monitor on the same host cannot report that host failing, and the outside check can.

### Rules

Expand All @@ -134,9 +149,42 @@ A running world without a rule's series becomes `unknown` after the hold window,
| `refusals` | world | any increase of `sim_world_refused_total` or `sim_world_shed_total` (fires at once; clears after 5 quiet minutes) | — |
| `stale-live` | world | `live.json` older than 30 s (fires at once) | — |
| `crash-loop` | any world | at least 3 non-planned restarts within 10 min (fires at once; clears after 10 min) | — |
| `viewers-high` | world | more than 1,000 viewer sockets on one world (fires at once). Information only: it never pages. | — |
| `launcher-loop-stall` | fleet | the launcher's own event-loop delay p99 > 0.25 s | 0.15 s |
| `worlds-created` | fleet | more than 10 worlds created in the last hour (fires at once). Counted across launcher restarts; creations before the collector's first scrape are not counted. | — |
| `disk-free` | fleet | under 20 % of the data disk free | — |
| `host-memory` | fleet | host memory available < 1 GiB | 1.5 GiB |
| `telemetry-gap` | collector | no answered scrape for more than 2.5 × the cadence. This covers failed scrapes, a silent collector seen by a later `alerts` run, and an empty store. | — |

Every rule pages except `refusals` and `viewers-high`, which are recorded in `alerts.jsonl` only. `refusals` fires on any refused request, which is the ceilings doing their job, not an incident.

### Reaching a person

With `--problems <data>/alert-problems.json`, the collector rewrites that file after every scrape. It holds `{at, everyMs, problems}`, with one problem per firing rule that pages, coded `alert:<rule>` (for example `alert:disk-free`).

- The launcher reads the file on every `/__sim/healthz`. A firing rule makes `hostOk` false and puts its code in `hostCodes` at once, and the code goes when the rule resolves.
- `deploy/host-health.mjs` (every 10 min) folds the same codes into `host-health.json` and logs an ALERT line in `host-health.log`. For an alert, healthz trusts the file it reads itself over this timer copy, so a resolved alert clears at once.
- A file older than 4 scrapes and at least 2 min means the collector died, and it shows as `alert:stale`. A clean stop (SIGTERM, `systemctl stop`) removes the file, so a stopped collector is never mistaken for a dead one.
- With no file (the collector not enabled), nothing changes.

The outside check watches `hostOk` on the public healthz, and that is how the alert leaves the machine. Setting it up is a hands-on step outside this repository.

The same health check also covers the hourly off-site backup. While `sim-worlds-backup-hourly@<stage>.timer` is enabled, a last run that did not succeed (a timeout, or a FAILED database, which exits 1) raises `backup-hourly:<stage>`, as the nightly run raises `backup:<stage>`. The owners' health page shows the failure, with the failed database from `backup-hourly-last.json` when the run wrote one.

### The unit

`deploy/sim-worlds-telemetry@.service` runs `telemetry.mjs collect --launcher-args --alerts --problems ...` as `simworlds`. It writes to `/srv/sim-worlds-data/<stage>/telemetry` and to `alert-problems.json` beside it. `--launcher-args` takes the launcher's port and base from `SIM_WORLDS_ARGS` in `/etc/sim-worlds/<stage>.conf`, so the unit needs no port of its own.

`install.sh` installs the unit on every release and never enables it. Enabling it is the maintainer's step, on the host:

```bash
systemctl enable --now sim-worlds-telemetry@production
systemctl status sim-worlds-telemetry@production
cat /srv/sim-worlds-data/production/alert-problems.json # {"at": ..., "problems": []} when all is well
```

To stop it, run `systemctl disable --now sim-worlds-telemetry@production`. The clean stop removes `alert-problems.json`.

### Changing the rules

`rules.json` overrides rules by id: `{"rules": {"loop-stall": {"threshold": 0.5, "holdMs": 60000}, "host-memory": {"off": true}}}`. The keys an override can set are `threshold`, `clearAt`, `holdMs`, `clearMs` and `off`. An unknown id is refused.
Expand All @@ -147,7 +195,7 @@ The launcher does not export process CPU, process RSS, cgroup CPU or memory pres

### Cost

With 400 registered worlds (20 running), evaluation after each scrape takes 40 ms at the median and at most 80 ms at the end of an hour. That is under 0.6 % of one core at the 15 s cadence. The state file is about 40 KB.
With 400 registered worlds (20 running), evaluation after each scrape takes 40 ms at the median and at most 80 ms at the end of an hour. That is under 0.6 % of one core at the 15 s cadence. The state file is about 40 KB. With the four spike rules (#2541), measured back to back against the rules before them over one synthetic hour (240 scrapes, two repeats each), the median was 40.8 ms against 39.0 ms and the maximum 76–79 ms against 73–80 ms.

## Cost and the context packet

Expand Down
24 changes: 16 additions & 8 deletions tools/world-launcher/deploy/host-health.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,17 @@
// short problem `codes` for everyone, the numbers for direct (loopback) requests. Alert when disk free < 15% OR < 20 GB,
// or MemAvailable < 1.5 GB; (t4-o1) when a sim-worlds@<stage> unit's working set (cgroup memory.current minus inactive
// file cache) reaches 90% of its MemoryMax, or its oom_kill count rose since the last run; and, only while a
// sim-worlds-backup@<stage>.timer is enabled, when that backup's last run did not succeed.
// No dependencies, no network. The off-box watcher (team/ops/sim-watch, Modal) reads `hostOk`/`hostCodes` from healthz and
// sim-worlds-backup@<stage>.timer is enabled, when that backup's last run did not succeed (#2541: likewise the hourly
// sim-worlds-backup-hourly@<stage>, code `backup-hourly:<stage>`); and (#2541) for each telemetry alert rule firing that
// pages, `alert:<rule>`, from <dataDir>/alert-problems.json (telemetry.mjs collect --problems), or `alert:stale` when
// the collector stopped writing that file without a clean stop.
// No dependencies (but telemetry/alerts.mjs, for the alerts file), no network. The off-box watcher (team/ops/sim-watch, Modal) reads `hostOk`/`hostCodes` from healthz and
// DMs Gene. Exit 0 always (a failed check must not flap the unit).
import { execFileSync } from 'node:child_process';
import { appendFileSync, existsSync, readdirSync, readFileSync, realpathSync, renameSync, statfsSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { readAlertProblems } from '../telemetry/alerts.mjs'; // #2541

export function memAvailableMb(meminfo) {
const m = /^MemAvailable:\s+(\d+)\s+kB/m.exec(meminfo);
Expand All @@ -29,7 +33,7 @@ export function readCgroup(dir) {
return { workingBytes: num('memory.current') - field(stat, 'inactive_file'), maxBytes: num('memory.max'), oomKills: field(events, 'oom_kill') };
}

export function evaluate({ diskTotalBytes, diskFreeBytes, memMb, units = {}, prevUnits = {}, backups = {}, worldProblems = [] }, t = {}) {
export function evaluate({ diskTotalBytes, diskFreeBytes, memMb, units = {}, prevUnits = {}, backups = {}, backupsHourly = {}, worldProblems = [], alertProblems = [] }, t = {}) {
const minPct = t.minFreePct ?? 15, minGb = t.minFreeGb ?? 20, minMem = t.minMemMb ?? 1536, capPct = t.capPct ?? 90;
const diskFreeGb = +(diskFreeBytes / 1e9).toFixed(1), diskFreePct = +(100 * diskFreeBytes / diskTotalBytes).toFixed(1);
const problems = [], codes = new Set();
Expand All @@ -49,9 +53,13 @@ export function evaluate({ diskTotalBytes, diskFreeBytes, memMb, units = {}, pre
for (const [stage, b] of Object.entries(backups)) {
if (b.enabled && b.result && b.result !== 'success') flag(`backup:${stage}`, `sim-worlds-backup@${stage} last run: ${b.result}`);
}
for (const [stage, b] of Object.entries(backupsHourly)) { // #2541: a timeout or a FAILED database fails the unit too
if (b.enabled && b.result && b.result !== 'success') flag(`backup-hourly:${stage}`, `sim-worlds-backup-hourly@${stage} last run: ${b.result}`);
}
for (const a of alertProblems) if (a?.code && a.text) flag(a.code, a.text);
// lane L1: the launcher's guard (children.mjs problems(), via <data>/world-problems.json): wedged, held, crashloop, bootSlow, halted
for (const w of worldProblems) if (w?.code && w.text) flag(w.code, w.text);
return { ok: problems.length === 0, diskFreeGb, diskFreePct, memAvailableMb: memMb, problems, codes: [...codes], units: unitStatus, backups,
return { ok: problems.length === 0, diskFreeGb, diskFreePct, memAvailableMb: memMb, problems, codes: [...codes], units: unitStatus, backups, backupsHourly,
thresholds: { minFreePct: minPct, minFreeGb: minGb, minMemMb: minMem, capPct } };
}

Expand All @@ -66,16 +74,16 @@ function readUnits(slice = CGROUP_SLICE) {
}

/** Backups count only while their timer is enabled (Gene, 2026-10-19: scheduled backups are off until he wants them). */
function readBackups(stages) {
function readBackups(stages, unit = 'sim-worlds-backup') {
const backups = {};
const show = (unit, props) => Object.fromEntries(execFileSync('systemctl', ['show', unit, '-p', props], { encoding: 'utf8', timeout: 10_000 })
.trim().split('\n').map(l => l.split('=')));
for (const stage of stages) {
try {
const timer = show(`sim-worlds-backup@${stage}.timer`, 'UnitFileState');
const timer = show(`${unit}@${stage}.timer`, 'UnitFileState');
if (timer.UnitFileState !== 'enabled') { backups[stage] = { enabled: false }; continue; }
// lane DASH: when the last run started and ended (the owner's health page shows them; '' = never ran)
const s = show(`sim-worlds-backup@${stage}.service`, 'Result,ExecMainStartTimestamp,ExecMainExitTimestamp,ExecMainStatus');
const s = show(`${unit}@${stage}.service`, 'Result,ExecMainStartTimestamp,ExecMainExitTimestamp,ExecMainStatus');
const iso = (v) => { const t = Date.parse(v ?? ''); return Number.isFinite(t) ? new Date(t).toISOString() : null; };
backups[stage] = { enabled: true, result: s.Result, startedAt: iso(s.ExecMainStartTimestamp), endedAt: iso(s.ExecMainExitTimestamp), exitStatus: s.ExecMainStatus === undefined ? null : Number(s.ExecMainStatus) };
} catch {}
Expand All @@ -101,7 +109,7 @@ function main(argv) {
const s = statfsSync(diskPath);
const units = readUnits();
status = evaluate({ diskTotalBytes: s.blocks * s.bsize, diskFreeBytes: s.bavail * s.bsize, memMb: memAvailableMb(readFileSync(`${process.env.SIM_PROC_ROOT ?? ''}/proc/meminfo`, 'utf8')),
units, prevUnits: prev?.units ?? {}, backups: readBackups(Object.keys(units)), worldProblems: readWorldProblems(dataDir) },
units, prevUnits: prev?.units ?? {}, backups: readBackups(Object.keys(units)), backupsHourly: readBackups(Object.keys(units), 'sim-worlds-backup-hourly'), worldProblems: readWorldProblems(dataDir), alertProblems: readAlertProblems(dataDir) },
{ minFreePct: Number(opt('min-free-pct', 15)), minFreeGb: Number(opt('min-free-gb', 20)), minMemMb: Number(opt('min-mem-mb', 1536)), capPct: Number(opt('cap-pct', 90)) });
} catch (e) { status = { ok: false, problems: [`check failed: ${e.message}`], codes: ['check'] }; }
status = { ...status, at: new Date().toISOString(), diskPath };
Expand Down
5 changes: 4 additions & 1 deletion tools/world-launcher/deploy/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -124,5 +124,8 @@ install -m 644 "$HERE/sim-worlds-backup@.service" /etc/systemd/system/sim-worlds
install -m 644 "$HERE/sim-worlds-backup@.timer" /etc/systemd/system/sim-worlds-backup@.timer
install -m 644 "$HERE/sim-worlds-health@.service" /etc/systemd/system/sim-worlds-health@.service
install -m 644 "$HERE/sim-worlds-health@.timer" /etc/systemd/system/sim-worlds-health@.timer
# #2541: the telemetry collector and its alerts. Installed, never enabled here (this runs on every release): enabling is
# the maintainer's step, docs/operations/telemetry.md.
install -m 644 "$HERE/sim-worlds-telemetry@.service" /etc/systemd/system/sim-worlds-telemetry@.service
systemctl daemon-reload
echo "installed: user simworlds, /srv/sim-worlds{,-data}, /etc/sim-worlds/{staging,production}.conf, sim-worlds@.service, sim-worlds-{backup,health}@.{service,timer} (nothing started)"
echo "installed: user simworlds, /srv/sim-worlds{,-data}, /etc/sim-worlds/{staging,production}.conf, sim-worlds@.service, sim-worlds-{backup,health}@.{service,timer}, sim-worlds-telemetry@.service (nothing started)"
27 changes: 27 additions & 0 deletions tools/world-launcher/deploy/sim-worlds-telemetry@.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# #2541: the telemetry collector for sim-worlds@<stage>: scrapes the launcher's loopback /__sim/metrics every 15 s into
# /srv/sim-worlds-data/<stage>/telemetry, evaluates the alert rules after each scrape, and writes the firing rules that
# page to alert-problems.json, which /__sim/healthz folds into hostOk/hostCodes (alert:<rule>) for the outside check.
# install.sh installs it and never enables it. Enabling is the maintainer's step (docs/operations/telemetry.md):
# systemctl enable --now sim-worlds-telemetry@production
[Unit]
Description=Telemetry collector and alerts for the sim-worlds@%i launcher
After=sim-worlds@%i.service

[Service]
Type=simple
User=simworlds
Group=simworlds
Nice=10
EnvironmentFile=/etc/sim-worlds/%i.conf
ExecStart=/usr/bin/node --no-warnings /srv/sim-worlds/%i/tools/world-launcher/telemetry.mjs collect --launcher-args --dir /srv/sim-worlds-data/%i/telemetry --alerts --problems /srv/sim-worlds-data/%i/alert-problems.json
Restart=always
RestartSec=10
MemoryMax=512M
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/srv/sim-worlds-data/%i
PrivateTmp=true

[Install]
WantedBy=multi-user.target
14 changes: 14 additions & 0 deletions tools/world-launcher/health-dash-backup.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,17 @@ test('the page says when the first record comes instead of "none"', () => {
assert.doesNotMatch(page, /tonight\\'s/);
new Function(page.split('<script>').at(-1).split('</script>')[0].replace(/^\s*import .*$/gm, '')); // the inline script still parses
});

test('#2541: a failed hourly backup, a FAILED database or a timeout, is an alert on the page', async () => {
const f = setup();
writeFileSync(join(f.dataDir, 'backup-hourly-last.json'), JSON.stringify({ at: isoAgo(0.5), ok: false, dbsOk: 6, dbs: 7, failed: ['worlds/alpha/sim.db: database disk image is malformed'] }));
writeFileSync(join(f.root, 'legacy', 'backup-hourly-last.json'), JSON.stringify({ at: isoAgo(2), ok: true, dbsOk: 7, dbs: 7 }));
const saved = host.backupsHourly;
host.backupsHourly = { legacy: { enabled: true, result: 'timeout' }, stale: { enabled: false, result: 'timeout' } };
try {
const texts = (await dashFor(f).json()).alerts.map((a) => a.text);
assert.ok(texts.includes('production hourly backup failed: worlds/alpha/sim.db: database disk image is malformed'), texts.join(' | '));
assert.ok(texts.includes('legacy hourly backup failed: timeout'), texts.join(' | '));
assert.ok(!texts.some((t) => /^stale hourly/.test(t)), 'a disabled hourly timer is not judged');
} finally { host.backupsHourly = saved; }
});
6 changes: 5 additions & 1 deletion tools/world-launcher/health-dash.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,9 @@ export function healthDash(o) {
const st = stages().map((s) => {
const k = s.self ? keeper : readJson(join(s.dir, 'keeper.json'));
return { stage: s.stage, self: s.self, build: s.self ? o.build : k?.active?.build ?? null, since: k?.active?.since ?? null,
backup: lastBackup(s.dir, s.stage, host?.backups?.[s.stage], backupRoot), timer: host?.backups?.[s.stage] ?? null };
backup: lastBackup(s.dir, s.stage, host?.backups?.[s.stage], backupRoot), timer: host?.backups?.[s.stage] ?? null,
// #2541: the hourly off-site run's own record and unit (a timeout leaves no record, only the unit's result)
hourly: readJson(join(s.dir, 'backup-hourly-last.json')), hourlyTimer: host?.backupsHourly?.[s.stage] ?? null };
});
const mem = memory();
const errs = o.errors.summary({ limit: 20 });
Expand Down Expand Up @@ -308,6 +310,8 @@ export function healthDash(o) {
}
for (const s of d.stages) {
if (s.backup && s.backup.ok === false) bad(`${s.stage} backup failed${s.backup.error ? `: ${String(s.backup.error).slice(0, 120)}` : ''}`);
const hu = s.hourlyTimer?.enabled && s.hourlyTimer.result && s.hourlyTimer.result !== 'success' ? s.hourlyTimer.result : null, hr = s.hourly?.ok === false ? s.hourly.error ?? s.hourly.failed?.[0] ?? 'see backup-hourly-last.json' : null;
if (hu || hr) bad(`${s.stage} hourly backup failed: ${String(hr ?? hu).slice(0, 120)}${hu && hr ? ` (unit: ${hu})` : ''}`);
// No backup seen at all (a new stage before its first night) is not an alert: the page says when the first record comes.
if (s.timer?.enabled && s.backup?.at && ago(s.backup.at) > 26) warn(`${s.stage}: no backup in ${Number.isFinite(ago(s.backup.at)) ? `${Math.round(ago(s.backup.at))} h` : 'its record'}`);
}
Expand Down
Loading
Loading