fake-detection: rebalance heuristic weights based on bounce-data signal analysis

[snackman]

Summary

First-round rebalance of backend/src/lib/fakeDetection.ts weights based on empirical signal-quality analysis run 2026-05-26 against:

• 9 confirmed-fraud events (>=15 scored emails AND >=30% Resend bounce/suppression rate)
• 194 confirmed-legit events (>=30 scored emails AND <=5% bad)

Ground truth = Resend bounce/suppression rate.

Four-tier summary

1. Up-weighted (3 heuristics)

Heuristic	Old	New	Empirical lift	Notes
wallet_too_low	8	12	+60%	Near-zero FP
pizzeria_fields_blank	5	10	+59%	Near-zero FP
low_funnel_coverage	10	15	+55%	1% FP rate

2. Disabled (4 heuristics removed from scoreEvent and WEIGHTS)

Heuristic	Lift	Why disabled
one_word_name	-21%	Fires on 88% of legit events (single-name cultures)
cross_event_wallet	-6%	Real crypto enthusiasts legitimately attend multiple events
wallet_reuse	-13%	Kiosk RSVPs legitimately share wallets
lsh_field_sig_cluster	+2%	65% FP rate; fires roughly equally on both classes

The exported checkXxx functions are preserved (with @deprecated JSDoc) so tests still validate them and we can re-enable with better thresholds later.

3. Audited (3 inert heuristics — never fired in either set)

Heuristic	Verdict	Recommendation
wallet_too_high_reuse	Dead in practice	LOWER-THRESHOLD-TO-X next PR. Requires walletRatio > 0.95 AND reuse > 0.3. Real events rarely exceed ~50% wallet adoption, so the >95% gate is unreachable. Suggest dropping walletRatio floor to ~0.5 or removing it and relying on reuse > 0.3 alone.
host_self_rsvp_mismatch	Correctly silent	KEEP-AS-IS. The situation it catches (someone RSVPs within 60s of event create under a non-host name) is rare but a near-certain fraud signal when it does fire. Weight 20 is appropriate; no change needed.
firstname_digits_email	Threshold too tight	LOWER-THRESHOLD-TO-X next PR. Requires ratio > 0.95 AND domainEntropy < 0.5 — even the most synthetic event datasets rarely hit 95% regex match. Suggest dropping ratio gate to ~0.7 and verifying signal quality before re-promotion.

4. Deliberately untouched (2 heuristics)

high_per_visitor_rsvp_saturation and repeat_session_rsvp_count are both at weight 20. Today's data suggests they're over-weighted, but the _rsvp_sid cookie is still relatively new — re-evaluate in ~3 weeks once more cookie-session data accumulates.

Calibration

• Before: 23 weights, total 245
• After: 19 weights, total 219
• Max score still capped at 100; tier cutoffs (10/30/60) unchanged.

Test plan

• cd backend && npx tsc --noEmit — no new errors in fakeDetection
• cd backend && npm test -- fakeDetection — all 108 tests pass
• (Optional) Re-run backend/scripts/heuristic-signal-quality.ts post-merge to confirm new totals look reasonable
• Spot-check /underboss fake-detection tab after merge to confirm previously-high-score events still surface

Follow-ups

• Re-evaluate high_per_visitor_rsvp_saturation and repeat_session_rsvp_count in ~3 weeks once cookie-session data accumulates.
• Next round: act on the three inert-heuristic verdicts above.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
rsvpizza	Ready	Preview, Comment	May 27, 2026 3:37am