Skip to content

plan.md

Latest commit

 

History

History
544 lines (383 loc) · 14.1 KB

plan.md

File metadata and controls

544 lines (383 loc) · 14.1 KB
Raw

Screen Architecture — 5 Screens, Bottom Tab Nav

Tab 1 — Home (Dashboard) Purpose: Daily health check at a glance Layout top to bottom:

Project selector dropdown (top bar) 3 metric cards row: Crashes Today / Affected Users / Active Sessions 7-day crash trend line chart "Recent Crashes" list

Each card: error name, node path, time ago, user count, 🎬 if replay available Color dot: red = new, yellow = seen, green = resolved

Tab 2 — Crashes Purpose: Full crash inbox, triage workflow Layout:

Filter bar: All / New / Resolved / Ignored Sort: Most recent / Most users / First seen Crash cards list

Error title

Obliesk Backend Plan

Backend plan for crash ingestion, replay storage, triage, alerts, billing, and team management.

1. Backend Goals

Build a backend that accepts SDK traffic from Godot games, normalizes crash and replay data, powers the dashboard, and reliably notifies teams when issues need attention.

Primary goals:

  • Ingest crashes and replay telemetry with low SDK overhead
  • Deduplicate, group, and enrich crash events
  • Support fast dashboard queries for projects, crashes, alerts, and replays
  • Deliver dependable notifications through email, push, and Discord
  • Support project, team, billing, and plan management
  • Keep the data model flexible enough for future features like traces, screenshots, and performance profiling

2. Scope

In scope

  • SDK ingestion API
  • Crash processing pipeline
  • Replay upload and retrieval
  • Projects, users, teams, and roles
  • Alert rules and alert events
  • Metrics aggregation for dashboard cards and trends
  • Share links for crashes and replays
  • Billing and subscription state
  • Audit logging for security-sensitive actions

Out of scope for v1

  • Full source map / script deobfuscation pipeline
  • Advanced distributed tracing
  • Long-term raw replay retention beyond plan limits
  • Custom on-prem deployment
  • Web client authentication federation beyond standard email and OAuth

3. Recommended Service Layout

The backend should start as a modular monolith with clear service boundaries, then split into workers and dedicated services where needed.

Core modules

  • API gateway / HTTP server
  • Auth and organization service
  • Project service
  • Ingestion service for SDK events
  • Processing workers for grouping and enrichment
  • Replay service
  • Alerts service
  • Billing service
  • Admin and audit service

Runtime components

  • Primary API server
  • Background job worker(s)
  • Queue / broker for async tasks
  • Relational database for source of truth
  • Object storage for replay blobs and attachments
  • Cache for hot dashboard reads and rate limiting
  • Email / push / webhook delivery integrations

4. System Architecture

Event flow

  1. SDK sends crash or replay payload
  2. API authenticates project DSN or signed token
  3. Raw payload is validated and stored durably
  4. A background job enriches, fingerprints, and groups the crash
  5. Aggregates are updated for dashboard cards and trends
  6. Alert rules are evaluated
  7. Notification jobs are queued and delivered
  8. Dashboard reads precomputed summaries and detail records

Separation of paths

  • Write path: ingestion, validation, persistence, job enqueueing
  • Read path: dashboard queries, crash details, replay playback, alert history
  • Control path: projects, users, billing, settings, alert configuration

5. Data Model

Use a relational database as the canonical store. Suggested core entities:

Identity and access

  • users
  • organizations
  • organization_members
  • roles
  • api_keys
  • sessions
  • audit_logs

Product structure

  • projects
  • project_members
  • project_settings
  • environments

Crash domain

  • crash_groups
  • crash_events
  • crash_occurrences
  • stack_frames
  • scene_snapshots
  • context_snapshots
  • breadcrumbs

Replay domain

  • replays
  • replay_frames
  • replay_snapshots
  • replay_assets

Alerts domain

  • alert_rules
  • alert_events
  • notification_channels
  • notification_deliveries

Billing and usage

  • subscriptions
  • plans
  • usage_counters
  • billing_invoices

Supporting tables

  • tags
  • project_tags
  • saved_views
  • share_links
  • webhook_endpoints

Key modeling decisions

  • crash_group is the user-facing issue; crash_event is the raw occurrence
  • Replays should be linked to crash occurrences, not just groups
  • Store hot dashboard aggregates separately from raw events
  • Keep raw payload JSON alongside normalized rows for forward compatibility

6. API Design

Public SDK APIs

  • POST /v1/sdk/crashes
  • POST /v1/sdk/replays
  • POST /v1/sdk/sessions/start
  • POST /v1/sdk/sessions/end
  • POST /v1/sdk/breadcrumbs

Dashboard APIs

  • GET /v1/projects
  • GET /v1/projects/:projectId/overview
  • GET /v1/projects/:projectId/crashes
  • GET /v1/crashes/:crashId
  • GET /v1/crash-groups/:groupId
  • GET /v1/replays
  • GET /v1/replays/:replayId
  • GET /v1/alerts/rules
  • GET /v1/alerts/events

Management APIs

  • POST /v1/projects
  • PATCH /v1/projects/:projectId
  • POST /v1/projects/:projectId/members
  • POST /v1/alerts/rules
  • PATCH /v1/alerts/rules/:ruleId
  • POST /v1/billing/checkout
  • POST /v1/billing/webhook

API design rules

  • Use versioned routes from day one
  • Return stable IDs and cursor-based pagination
  • Separate write payload validation from read-model serialization
  • Keep SDK endpoints tolerant to partial payloads and schema evolution
  • Use idempotency keys for retry-safe uploads

7. Ingestion Pipeline

Crash ingestion

  • Validate DSN or project token
  • Normalize platform, environment, build version, and device metadata
  • Persist raw payload first
  • Compute fingerprint from error name, top frames, node path, and scene path
  • Map into or create a crash_group
  • Increment occurrence counts and affected-user counts
  • Update first-seen / last-seen timestamps

Replay ingestion

  • Accept replay metadata and frame stream separately or as a bundled upload
  • Compress and store large payloads in object storage
  • Associate replay with session, crash occurrence, and project
  • Generate a lightweight preview record for list views

Enrichment jobs

  • Parse stack traces into structured frames
  • Normalize platform labels
  • Derive scene names and node path hints
  • Attach alert evaluation inputs
  • Generate dashboard aggregates

8. Crash Grouping Strategy

The backend needs stable grouping so the dashboard feels trustworthy.

Fingerprint inputs

  • Error class or message
  • Top stack frame function and file
  • Node path
  • Scene path
  • Platform if the issue is platform-specific

Grouping rules

  • Start with deterministic rules, not ML
  • Preserve manual override capability for merges and splits
  • Recompute fingerprints when symbolication or normalization improves
  • Store both fingerprint version and group version for future changes

Triage statuses

  • new
  • seen
  • resolved
  • ignored

Status history should be auditable rather than overwritten silently.

9. Replay Service

Responsibilities

  • Store and retrieve replay timelines
  • Keep preview metadata for list views
  • Serve data for the replay viewer scrubber, markers, and overlays
  • Support access control for share links and team members

Storage approach

  • Small replay metadata in the database
  • Large frame payloads in object storage
  • Optional precomputed thumbnails or preview manifests

Playback requirements

  • Query by replay ID and time range
  • Support marker types: scene, signal, input, crash
  • Support speed controls and frame stepping in the client
  • Deliver compact payloads so the dashboard stays responsive

10. Alerts and Notifications

Rule types

  • Crash count threshold
  • New crash group
  • Platform-specific crash
  • Project-specific or global rules

Delivery channels

  • Email
  • Push
  • Discord webhook

Alert flow

  1. Crash or aggregate changes arrive
  2. Rules are evaluated asynchronously
  3. Matching rules create alert events
  4. Delivery jobs fan out to channels
  5. Delivery status is tracked per attempt

Anti-noise rules

  • Debounce repeated notifications for the same group
  • Respect quiet hours where configured
  • Cap notifications during incident storms
  • Allow per-project suppression rules

11. Auth and Access Control

Authentication

  • Email/password or passwordless for dashboard users
  • DSN or signed project token for SDK traffic
  • Session cookies or bearer tokens for dashboard clients

Authorization

  • Organization-level roles: owner, admin, member
  • Project-level permissions for read/write operations
  • Strict access checks for replay playback and share links
  • Separate internal admin access from customer access

Security controls

  • Rate limiting per DSN, IP, and token
  • Payload size limits
  • Schema validation and content type enforcement
  • Audit logs for membership, billing, and rule changes

12. Billing and Usage

Billing objects

  • Plan tiers: Free, Indie, Studio
  • Project quotas: events, retention, alert rules, team members
  • Metered usage: crash events, replay storage, notification volume

Enforcement

  • Enforce limits at ingestion and storage layers
  • Surface soft warnings before hard caps are reached
  • Keep billing state separate from crash processing so ingestion remains resilient

Provider integration

  • Use a billing provider webhook for subscription lifecycle updates
  • Keep invoice and entitlement records locally for fast reads

13. Observability and Operations

Required telemetry

  • Request latency and error rate
  • Queue lag and job failures
  • Ingestion throughput by project
  • Notification delivery success and retries
  • Storage growth by project and environment

Operational tooling

  • Dead-letter queue for failed jobs
  • Admin replay of ingestion payloads for debugging
  • Manual resend for notifications
  • Feature flags for rollout control

SLO targets

  • Ingestion acknowledgment: fast enough for SDK retries to stay rare
  • Dashboard read latency: sub-second for common queries
  • Notification delivery: near-real-time for new crash alerts

14. Non-Functional Requirements

  • Idempotent writes for SDK retries
  • Backward-compatible payload evolution
  • Multi-tenant isolation by project and organization
  • GDPR-friendly data retention controls
  • Predictable cost growth with retention and replay volume
  • Graceful degradation if notification providers fail

15. Implementation Phases

Phase 1 — Backend MVP

  • Project and organization schema
  • SDK ingestion endpoint for crash events
  • Crash grouping and deduplication
  • Dashboard read APIs for overview and crash list
  • Basic replay metadata support
  • Auth for dashboard users and project tokens
  • Minimal alert rule storage

Phase 2 — Reliability and Triage

  • Background job queue
  • Retry-safe ingestion and idempotency keys
  • Enrichment pipeline for stack traces and metadata
  • Crash detail APIs with scene tree and context snapshots
  • Status transitions: new, seen, resolved, ignored
  • Audit logging for triage actions

Phase 3 — Replay and Alerts

  • Full replay frame upload and storage
  • Replay retrieval APIs and share links
  • Alert evaluation engine
  • Notification delivery integrations
  • Notification history and retry tracking

Phase 4 — Billing and Scale

  • Subscription and plan enforcement
  • Usage metering
  • Retention policies by plan
  • Cache layer for hot dashboards
  • Object storage lifecycle rules
  • Rate-limit hardening and abuse detection

Phase 5 — Advanced Backend Features

  • Attachments and screenshots
  • Symbolication / source map support
  • Team activity feed
  • Public status and incident pages
  • Webhook events for customer integrations
  • Export jobs for analytics and compliance

16. Suggested Tech Stack

The exact stack can vary, but the backend should support these capabilities cleanly.

  • API server: Node.js, TypeScript, or Go
  • Database: PostgreSQL
  • Cache: Redis
  • Queue: Redis queue, SQS, or RabbitMQ
  • Object storage: S3-compatible storage
  • Deployment: containerized services
  • Monitoring: OpenTelemetry-compatible tracing and metrics

If the team wants the fastest path, a TypeScript monolith with a worker process is the simplest starting point.

17. Risks and Open Questions

  • Replay storage cost may grow quickly without strong retention rules
  • Crash grouping needs careful tuning to avoid noisy duplicates or over-merging
  • SDK retry behavior must be matched by backend idempotency
  • Alert delivery can become noisy without debounce and suppression logic
  • Billing limits need to be enforced without blocking critical crash ingestion
  • Privacy policy needs to define what SDK context data is allowed by default

18. Testing Without Real Games

The backend can be built and validated before any production game integration exists.

Test inputs

  • Synthetic crash payloads that match the SDK schema
  • Replay fixtures with marker timelines and scene snapshots
  • Contract tests that replay captured request/response examples
  • Load tests using generated crash bursts and replay uploads

Minimal validation harness

  • A tiny Godot sample project that only exercises init, crash capture, and replay markers
  • A CLI payload generator for API smoke tests
  • Postman or HTTP client collections for manual endpoint checks
  • Golden files for crash grouping and dashboard aggregate calculations

What to verify first

  • Ingestion accepts valid and invalid payloads correctly
  • Crash grouping is stable across repeated submissions
  • Replay uploads are stored and retrievable
  • Alert rules fire from synthetic crash thresholds
  • Dashboard read APIs return consistent summaries from seeded data

Outcome

This keeps backend development unblocked even if there are no real games available yet, and it reduces risk before SDK integration lands.

19. Immediate Next Steps

  • Define the public API contract for SDK ingestion and dashboard reads
  • Choose the initial stack and deployment target
  • Finalize the relational schema for projects, crashes, replays, and alerts
  • Implement crash ingestion and grouping first
  • Add replay storage second
  • Add notifications and billing after the core read/write flow is stable