Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 39 additions & 5 deletions docs/operations/cdn.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,9 +227,41 @@ the launcher keyed that request on (an entry that is not an address answers `nul
the `ip=` line of `/cdn-cgi/trace` on the same name), `forged` (the same with a forged `X-Forwarded-For`), `viewer`
(a viewer WebSocket gets its welcome), `agent` (a request with no browser headers to `/__sim/healthz` and a token-less
`POST /w/<world>/v1/edge/act` get the launcher's JSON, not a challenge), `events` (the first event of
`/w/<world>/v1/edge/events` within 5 s) and `ranges` (the snippet's ranges equal the published lists). It prints each
problem and PASS or FAIL. Tests: `node --test tools/world-launcher/check-soul-game.test.mjs` (A1/A2 need `caddy` on
`PATH` or in `$CADDY`).
`/w/<world>/v1/edge/events` within 5 s), `ranges` (the snippet's ranges equal the published lists) and `cache` (the
cache rule below: the world page's script bundle, its first snapshot part, a release-stable file, `--stable`, default
`/mars-assets/draco/draco_decoder.wasm`, and the snapshot under the page's own key, each requested twice with the
second a `HIT`; the snapshot under a stale key is `no-store` and never a hit). It prints each problem and PASS or FAIL;
`--points cache` runs one point. Edge caches are per site, so a hit here says nothing about other locations. Tests:
`node --test tools/world-launcher/check-soul-game.test.mjs` (A1/A2 need `caddy` on `PATH` or in `$CADDY`) and
`node --test tools/cdn/origin-cache.test.mjs` (the rule, and the `cache` point through a stand-in CDN).

### The cache rule

A CDN caches by file extension by default and never JSON or extension-less paths, so the content-addressed parts,
profile photos and the snapshot would read `DYNAMIC`, and the release's `cdn-walk` fails any hashed or release-stable
response that does (`tools/cdn/origin-cache.mjs`). One cache rule in the `soul.game` zone fixes that. Its paths are
`CDN_CACHE_RULE` in `origin-cache.mjs`, derived from the same classes the release checks plus the keyed snapshot; the
test checks that this page carries `cdnRuleExpression()` exactly:

```
http.host eq "soul.game" and ((http.request.uri.path strict wildcard "/w/*/v1/world/part/*" or http.request.uri.path strict wildcard "/w/*/v1/souls/*/profile-photo/*" or http.request.uri.path strict wildcard "/w/*/v1/world/snapshot") or ((http.request.uri.path strict wildcard "/assets/*" or http.request.uri.path strict wildcard "/w/*/assets/*" or http.request.uri.path strict wildcard "/w/*/art/*" or http.request.uri.path strict wildcard "/modular/*" or http.request.uri.path strict wildcard "/mars-assets/*" or http.request.uri.path strict wildcard "/world-assets/*" or http.request.uri.path strict wildcard "*.wasm" or http.request.uri.path strict wildcard "*.woff" or http.request.uri.path strict wildcard "*.woff2" or http.request.uri.path strict wildcard "*.ttf" or http.request.uri.path strict wildcard "*.otf") and not http.request.uri.path contains "/v1/" and not http.request.uri.path contains "/__sim/"))
```

| Setting | Value |
|---|---|
| Cache eligibility | Eligible for cache |
| Edge TTL | Use cache-control header if present, bypass cache if not |
| Browser TTL | Respect origin TTL |
| Cache key | The default (the query string is part of it: the snapshot's `v=` key is) |

The rule only makes these paths eligible; the origin's `Cache-Control` decides. Bundles, stored objects, parts, art and
photos are `immutable` for a year, release-stable files keep an hour or more, and the snapshot is
`public, max-age=0, s-maxage=86400` only when `?v=` is the world's current key (`apps/server/src/edge-snapshot.ts`),
`no-store` otherwise, so a stale or invented key never enters the cache. A `no-store`, `private` or header-less answer
on an eligible path is never kept. No path that can answer per user is eligible: the static patterns never match under
`/v1/` or `/__sim/`, and the three API reads are public by design (moderation-filtered, no auth read). The page,
`/v1/config` and the world list stay outside it. Add no rule that bypasses everything else: other paths keep the CDN's
default, which also respects the origin's headers.

### Hands-on steps, in this order

Expand Down Expand Up @@ -267,15 +299,17 @@ The release never writes Caddy's configuration. Do these once, after a release t
INSTALL
```

Before the record is proxied, the check fails only `address` (no trace page yet); `forged` must already pass.
Before the record is proxied, the check fails only `address` (no trace page yet) and `cache` (no cache status);
`forged` must already pass.
2. **Turn off bot challenges and script rewriting** in the CDN zone: no Bot Fight Mode, managed challenge, Browser
Integrity Check or challenge rule on `/v1/*`, `/w/*/v1/*`, `/__sim/*` or the page, and no Rocket Loader, Email
Obfuscation, minification or injected scripts on any path. Agents' command-line clients, the release's health
checks, the walk and eyes runners and the off-box watcher are not browsers, and the page depends on its inline
script order.
3. **Keep the sign-in records unproxied** (DNS only): Clerk's records for the domain must answer from Clerk.
4. **TLS mode Full (strict)**: the CDN connects to the origin over HTTPS and checks Caddy's certificate.
5. **Switch the `soul.game` record to proxied, together with E2's cache rules** (never one without the other).
5. **Switch the `soul.game` record to proxied, together with [the cache rule](#the-cache-rule)** (never one without
the other: without the rule the next release's `cdn-walk` fails).
6. **Soak**: run `check-soul-game.mjs` from outside until it prints PASS, then keep it and the release's health checks
running through a busy period. The CDN restarts sockets without notice, so viewer bars are measured at the origin.
7. **Last, restrict the firewall** so ports 80 and 443 accept only the ranges in `cdn-trust.snippet`. Leave SSH as it
Expand Down
9 changes: 9 additions & 0 deletions docs/operations/runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,15 @@ To put a world's data back live: stop the stage, move the world's `sim.db` aside
`chown simworlds:simworlds`, start the stage. Restoring data discards events accepted since the snapshot, so do it only
for real data loss. Tests: `node --test tools/world-launcher/backup-restic.test.mjs` (skips without restic).

**CDN purges** (behind the CDN, [cache rule](cdn.md#the-cache-rule)): parts, uploads and art are immutable for a year at
the edge, and the origin's later refusal never reaches copies already there. Purge by prefix `soul.game/w/<slug>/` where
the plan has it, otherwise purge everything:

- **When moderation vetoes a file**, purge its URLs (`/w/<slug>/assets/<sha>.<ext>`, `/w/<slug>/art/<sha>.<ext>`) and
that world's parts and snapshot.
- **After a restore** of a world's data, purge that world: its snapshot key can repeat a key the edge already holds for
other contents.

Release backups (`ship.sh` step `backup`) are separate: row-verified compact copies in
`/srv/sim-worlds-data/release-backups/pre-r<label>-<stage>-<date>-<time>-XXXXXX` (a random six-character suffix; older
copies have none). `deploy/retention.sh` lists and prunes old ones, newest first by the date and time in the name, dry
Expand Down
31 changes: 31 additions & 0 deletions tools/cdn/origin-cache.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,34 @@ export function originCacheProblem({ path, status, headers }) {
if ((cf === 'DYNAMIC' || cf === 'BYPASS') && !revalidated) return `Cloudflare ${cf} for an origin ${cls} asset: ${path}`;
return null;
}

// The CDN cache rule for soul.game (E2): the paths a shared cache may keep, derived from the classes above (in their
// root-launcher form; soul.game has no /worlds prefix) plus the keyed snapshot (apps/server/src/edge-snapshot.ts). The
// rule only makes a path eligible; the origin's Cache-Control decides (edge TTL "use the header if present, bypass if
// not"), so a no-store, private or header-less answer on an eligible path is never kept. No path that can answer per
// user is listed: the static patterns never match under /v1/ or /__sim/, and the three API reads are public by design
// (moderation-filtered, no auth read). docs/operations/cdn.md carries cdnRuleExpression() verbatim (the test checks).
export const CDN_CACHE_RULE = {
host: 'soul.game',
// static files: Vite bundles and content-addressed objects (HASHED 1, 2, 4), release-stable files (STABLE)
files: ['/assets/*', '/w/*/assets/*', '/w/*/art/*', '/modular/*', '/mars-assets/*', '/world-assets/*', '*.wasm', '*.woff', '*.woff2', '*.ttf', '*.otf'],
// public reads: snapshot parts (HASHED 3), profile photos (HASHED 5), the snapshot (s-maxage only for the current ?v= key)
api: ['/w/*/v1/world/part/*', '/w/*/v1/souls/*/profile-photo/*', '/w/*/v1/world/snapshot'],
never: ['/v1/', '/__sim/'], // no static pattern matches a path containing these
};
// The CDN's `strict wildcard`: `*` is any run of characters, slashes included; the rest is literal and case-sensitive,
// matched against the whole path without its query string.
const wild = (pattern) => new RegExp('^' + pattern.split('*').map((s) => s.replace(/[.+?^${}()|[\]\\]/g, '\\$&')).join('.*') + '$');
const RULE_FILES = CDN_CACHE_RULE.files.map(wild), RULE_API = CDN_CACHE_RULE.api.map(wild);
/** Whether the documented rule makes a path eligible for the CDN's cache (the origin's headers still decide). */
export function cdnCacheEligible(path) {
const p = path.split('?')[0];
if (RULE_API.some((r) => r.test(p))) return true;
return RULE_FILES.some((r) => r.test(p)) && !CDN_CACHE_RULE.never.some((s) => p.includes(s));
}
/** The rule as the CDN's filter expression, exactly as docs/operations/cdn.md gives it. */
export function cdnRuleExpression() {
const any = (list) => list.map((x) => `http.request.uri.path strict wildcard "${x}"`).join(' or ');
const not = CDN_CACHE_RULE.never.map((x) => `not http.request.uri.path contains "${x}"`).join(' and ');
return `http.host eq "${CDN_CACHE_RULE.host}" and ((${any(CDN_CACHE_RULE.api)}) or ((${any(CDN_CACHE_RULE.files)}) and ${not}))`;
}
87 changes: 86 additions & 1 deletion tools/cdn/origin-cache.test.mjs
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { originCacheClass, originCacheProblem } from './origin-cache.mjs';
import { createServer } from 'node:http';
import { readFileSync } from 'node:fs';
import { cdnCacheEligible, cdnRuleExpression, originCacheClass, originCacheProblem } from './origin-cache.mjs';
import { checkSoulGame } from '../world-launcher/deploy/caddy/check-soul-game.mjs';
import { EDGE_SNAPSHOT_CACHE } from '../../apps/server/src/edge-snapshot.ts';

const IMM = 'public, max-age=31536000, immutable', DAYSWR = 'public, max-age=86400, stale-while-revalidate=604800';
const r = (path, cc, cf = 'HIT', status = 200) => ({ path, status, headers: { ...(cc ? { 'cache-control': cc } : {}), ...(cf ? { 'cf-cache-status': cf } : {}) } });
Expand Down Expand Up @@ -55,3 +59,84 @@ test('legacy core GLBs may revalidate while uploaded GLBs and other hash objects
assert.match(originCacheProblem(r(png, VIEW, 'DYNAMIC')), /immutable/);
assert.equal(originCacheProblem(r(png, IMM, 'HIT')), null);
});

// --- E2: the CDN cache rule ------------------------------------------------------------------------------------------
// One sample path per hashed and stable class, in soul.game's root-launcher form, with the policy the origin sends it.
const RULE_SAMPLES = [
['/assets/LiveWorld-BN26MIPk.js', IMM], // Vite bundle
[`/assets/${SHA}.png`, IMM], [`/w/eden/assets/${SHA}.png`, IMM], // content-addressed stored objects
['/w/eden/v1/world/part/8cd8bf3068833a807af5426e595f3290', IMM], // snapshot part
[`/w/eden/art/${SHA}.png`, IMM], // hung art
['/w/eden/v1/souls/s_1/profile-photo/0123456789abcdef', IMM], // profile photo
['/modular/humanoid/kit.json', DAYSWR], ['/mars-assets/draco/draco_decoder.wasm', DAYSWR], ['/world-assets/eden/sky.json', DAYSWR],
['/w/eden/fonts/inter.woff2', DAYSWR],
];
test('E2 A1: every hashed and stable class is eligible under the documented rule; HIT/MISS accepted, DYNAMIC/BYPASS refused', () => {
const seen = new Set();
for (const [path, cc] of RULE_SAMPLES) {
const cls = originCacheClass(path); assert.ok(cls === 'hashed' || cls === 'stable', `${path}: ${cls}`); seen.add(cls);
assert.equal(cdnCacheEligible(path), true, `${path} is not eligible under the rule`);
for (const cf of ['HIT', 'MISS', 'EXPIRED', 'REVALIDATED']) assert.equal(originCacheProblem(r(path, cc, cf)), null, `${path} ${cf}`);
for (const cf of ['DYNAMIC', 'BYPASS']) assert.match(originCacheProblem(r(path, cc, cf)), new RegExp(`Cloudflare ${cf}`), `${path} ${cf}`);
}
assert.deepEqual([...seen].sort(), ['hashed', 'stable']);
assert.equal(cdnCacheEligible('/w/eden/v1/world/snapshot?defs=ref&cols=1&v=12.3.4'), true, 'the snapshot (the origin keeps it no-store unless keyed)');
// nothing that can answer per user, nor the page and the lists (E3's)
for (const p of ['/', '/w/eden/', '/w/eden', '/v1/config', '/w/eden/v1/config', '/v1/site/worlds', '/__sim/healthz', '/__sim/thumbs/eden.webp', '/w/eden/ws', '/w/eden/v1/edge/act',
'/w/eden/v1/edge/events', `/w/eden/v1/god/assets/${SHA}`, '/w/eden/v1/god/fonts/x.woff2', '/w/eden/v1/art/media', '/v1/me', `/w/eden/asset-view/${SHA}.glb`]) assert.equal(cdnCacheEligible(p), false, p);
});
test('E2 A3: docs/operations/cdn.md gives the rule expression verbatim', () => {
const doc = readFileSync(new URL('../../docs/operations/cdn.md', import.meta.url), 'utf8');
assert.ok(doc.includes(cdnRuleExpression()), 'cdn.md does not carry cdnRuleExpression(): paste the current one');
});

const listen = (s) => new Promise((yes) => s.listen(0, '127.0.0.1', () => yes({ url: `http://127.0.0.1:${s.address().port}`, close: () => { s.closeAllConnections(); s.close(); } })));
/** A stand-in CDN: keeps a GET only on a path `eligible` allows, for the origin's s-maxage or max-age (none, no-store,
* private or 0: not kept, BYPASS); anything else passes through as DYNAMIC. Cloudflare's cf-cache-status words. */
function standInCdn(origin, eligible) {
const kept = new Map();
return listen(createServer(async (req, res) => {
const send = (st, h, body, cf, extra = {}) => res.writeHead(st, { ...h, 'cf-cache-status': cf, ...extra }).end(body);
const hit = kept.get(req.url);
if (eligible(req.url) && hit && hit.until > Date.now()) return send(hit.status, hit.headers, hit.body, 'HIT', { age: String(Math.floor((Date.now() - hit.at) / 1000)) });
const o = await fetch(origin + req.url, { headers: { accept: req.headers.accept ?? '*/*' } }), body = Buffer.from(await o.arrayBuffer());
const cc = o.headers.get('cache-control') ?? '', headers = { 'content-type': o.headers.get('content-type') ?? '', ...(cc ? { 'cache-control': cc } : {}) };
if (!eligible(req.url)) return send(o.status, headers, body, 'DYNAMIC');
const ttl = Number((/s-maxage=(\d+)/.exec(cc) ?? /max-age=(\d+)/.exec(cc) ?? [0, 0])[1]);
if (o.status !== 200 || !cc || /no-store|private/.test(cc) || ttl <= 0) return send(o.status, headers, body, 'BYPASS');
kept.set(req.url, { status: o.status, headers, body, at: Date.now(), until: Date.now() + ttl * 1000 });
send(o.status, headers, body, 'MISS');
}));
}
/** The origin as the launcher answers these paths: the page (never cached) with the snapshot key and parts, the bundle,
* a part, a stable file, and the snapshot (cacheable only under the current key, as edge-snapshot.ts does). */
function standInOrigin({ staleCacheable = false } = {}) {
const KEY = '41.2.7', PART = 'ab'.repeat(16);
return listen(createServer((req, res) => {
const [path, q = ''] = req.url.split('?'), v = new URLSearchParams(q).get('v');
const ok = (type, cc, body) => res.writeHead(200, { 'content-type': type, 'cache-control': cc }).end(body);
if (path === '/w/eden/') return ok('text/html', 'no-cache', `<!doctype html><head><script>window.__simSnapshotKey="${KEY}";window.__simSnapshotParts=["${PART}","${'cd'.repeat(16)}"]</script><script type="module" crossorigin src="/assets/index-AbCd1234.js"></script></head>`);
if (path === '/assets/index-AbCd1234.js') return ok('text/javascript', IMM, 'export {}');
if (path === `/w/eden/v1/world/part/${PART}`) return ok('application/json', IMM, '{}');
if (path === '/mars-assets/draco/draco_decoder.wasm') return ok('application/wasm', 'public, max-age=86400', 'wasm');
if (path === '/w/eden/v1/world/snapshot') return ok('application/json', v === KEY || staleCacheable ? EDGE_SNAPSHOT_CACHE : 'no-store', '{"tick":1}');
res.writeHead(404).end();
}));
}
const cacheCheck = async (eligible, opts) => {
const o = await standInOrigin(opts), cdn = await standInCdn(o.url, eligible);
try { return await checkSoulGame({ origin: cdn.url, points: ['cache'], waitMs: 3000 }); } finally { cdn.close(); o.close(); }
};
test('E2 A2: the check\'s cache section passes through a stand-in CDN configured from the documented rule', async (t) => {
const r = await cacheCheck(cdnCacheEligible);
t.diagnostic(`check-soul-game --points cache:\n${r.seen.map((x) => ` ${x}`).join('\n')}\n${r.pass ? 'PASS' : 'FAIL'}`);
assert.equal(r.pass, true, r.problems.join('\n'));
assert.equal(r.seen.length, 5, r.seen.join('\n'));
});
test('E2 A2: the same check fails without the rule (the CDN\'s by-extension default) and when a stale key is kept', async () => {
const byExtension = (u) => /\.(?:js|css|png|jpg|wasm|woff2?)$/.test(u.split('?')[0]);
const d = await cacheCheck(byExtension);
assert.deepEqual(d.problems.map((p) => /^cache: (\w+)/.exec(p)?.[1]), ['part', 'snapshot'], d.problems.join('\n'));
const s = await cacheCheck(cdnCacheEligible, { staleCacheable: true });
assert.equal(s.problems.length, 1, s.problems.join('\n')); assert.match(s.problems[0], /stale key answered HIT/);
});
Loading
Loading