feat(backend): implement data retention policies

[Darkdruce]

Closes #98

Implements the data retention subsystem in backend/database/retention/, covering all six items in the issue.

What's here

Issue item	Implementation
Automatic data archival	archiver.ts — batched sweep to cold storage
Data purging	archiver.ts — verified delete, keyset-paginated
Compliance tracking	compliance-log.ts — hash-chained append-only audit trail
Archive retrieval	restore.ts, manifest.ts — searchable manifest + checksum-verified restore
Data anonymization	anonymizer.ts — keyed HMAC pseudonymization / redaction
Retention monitoring	monitor.ts — health snapshot, severity-ranked alerts, Prometheus gauges

Storage backends are pluggable (ColdStorageBackend); a filesystem implementation is included for local/test use and an S3 Glacier one for production.

Design notes worth reviewing

The sweep's ordering is the safety argument. Every step is durable before the destructive one runs, so a crash at any point leaves data duplicated (recoverable) rather than lost. Concretely: archive → read back → verify checksum and row count → record manifest → delete → verify deletion. An object that cannot be retrieved and verified is not an archive, and deleting against it would be data loss dressed up as compliance.

PII never reaches cold storage. Rows are scrubbed before serialization, not after. Stellar addresses are pseudonymized with a keyed HMAC rather than a plain digest — the address space is small enough that an unkeyed SHA-256 is brute-forceable, which would leave the "anonymized" data still personal under GDPR.

The audit trail is defensible, not just present. Entries commit to their predecessor's hash, so an edit or deletion invalidates every hash after it, and an auditor can verify independently. The table additionally carries a trigger rejecting UPDATE/DELETE — without it, an operator with table privileges could rewrite history and re-chain the hashes.

Policies are the injection surface. Table and column names are interpolated into SQL where bind parameters aren't allowed, so validatePolicy rejects anything that isn't a bare lowercase identifier.

Monitoring reports state, not the last run. A sweep returning clean results every time is worthless if it stopped being scheduled a month ago, and only a standing check notices that. Backlog counts are capped at 100k via a LIMIT inside a subquery so the health check doesn't get slower exactly as the situation gets worse. Stale-sweep severity follows the regime: on a GDPR class it's a compliance breach, not a backlog.

Tests

33 tests in backend/tests/database/retention.test.ts, all passing. They run against an in-memory fake pg pool rather than mocks — the safety argument is about the ordering of reads, writes and deletes, and only something stateful can tell whether rows were still present when the delete ran.

Coverage is aimed at the failures that are silent and expensive: deleting against a corrupted archive, PII surviving into cold storage, a tampered audit trail that still verifies, a stalled sweep nothing alerts on.

The tests caught a real bug during development: deletion was bounded by timestamp < rangeEnd, but rangeEnd is the max timestamp in the batch — so the newest row of every batch was archived and then never deleted, sitting duplicated in both tiers permanently since the keyset cursor had already moved past it. Fixed to <=, which preserves the original intent of sparing a row updated mid-flight.

Schema

Migration 1724000000000_add-retention-tables.js and the matching db/schema.sql block. The integration suite bootstraps from schema.sql, so changing one without the other would let tests pass against a shape production never has — the constraints are duplicated in both deliberately.

Note for reviewers

backend/tsconfig.json has include: ["src"], but there is no backend/src/ directory, and jest.config.json sets diagnostics: false. Nothing in backend/ is currently typechecked, including this module. I typechecked it explicitly (tsc --strict, clean) and that surfaced two latent type errors in code I'd otherwise have shipped unchecked. Widening include would pull in every root-level backend/*.ts file at once, so I've left it alone as a separate decision — but it's worth making.