docs: add SaaS platform plan as living doc - #42
Open
impressiver wants to merge 11 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Captures the auth provider evaluation, initial v0 plan, adversarial panel review (PM/Engineer/CEO), and the BYOK-vs-managed-inference resolution. Includes phased build, day-1 must-gets, deferrals, compliance posture, and open questions. Document is structured as decision history + current plan so future revisions append rationale rather than overwriting context. Maintenance conventions documented at the top.
Adds §1.5 Postgres minimization directive to the decision history and propagates the implications through §2: - bucket_state moves to Redis (Upstash) with atomic Lua reserve/settle; 5-minute reconciler against Tinybird UsageEvent sum is the durability backstop - audit_log moves to Axiom, shipped from a Postgres outbox; immutability is structural in the log store - Idempotency-Key fingerprint cache and WorkOS membership cache move to Redis - bucket_periods historical detail stays in Tinybird; only the per-period billing_period_summary row lives in Postgres Postgres now holds only the small joined transactional core (accounts, members, api_keys, buckets-config, billing_period_summary, outbox). No hot-path counters, no append-only event tables, no caches.
Adds §3 proposing seven per-purpose specs in four priority tiers, with explicit dependency graph and naming convention: - Tier 1 (foundation): S1 Identity & Access, S2 Metering Spine - Tier 2 (wedge): S3 API Keys, S4 Inference Gateway & SDK - Tier 3 (monetization): S5 Billing & Credits - Tier 4 (activation/trust): S6 Activation & Pricing, S7 Trust & Compliance Specs would live at specs/saas/NN-*.md (sibling to the framework specs 01–22, not interleaved). Follows the existing spec-guidelines.md convention: ideal state only, no phased language inside specs. S1 and S2 are the contract surface for everything else; recommend locking those before drafting S3–S7. S3/S5 can be drafted in parallel once S2 is stable.
§1.6 decision-history entry capturing Unkey's mid-2026 pricing (150K free, $25 base, no overages), the 6M-verification/month free-cohort math that blows through the free tier, and the structural answer: verification caching collapses the vendor-vs-in-house gap to ~0, and once present makes in-house structurally cheaper (no hot-path latency, no vendor in auth path, no custody concerns). §2.13 new "Cost-coverage invariants" — six architectural constraints (Redis cache, per-key RPS limit, daily caps, BYOK default, $5 trial pool, single Upstash instance) that bound free-user marginal cost to fractions of a cent/month and survive vendor swaps. §2.5 stack table: API keys now lock in-house with the cache spec. §2.6 P2 phase: 3–4 days, not a week; verification cache + per-key RPS limit added explicitly. §2.11: Unkey-vs-custom open question resolved (struck through with reference to §1.6).
impressiver
force-pushed
the
docs/saas-platform-plan
branch
from
June 14, 2026 22:41
c870b45 to
13b5e46
Compare
§1.7 decision-history entry: Cloudflare for compute/edge (Workers, Pages, R2, AI Gateway, Hyperdrive, Cron, Queues, Secrets); three external vendors fill gaps Cloudflare doesn't offer (Neon, Upstash, Axiom). No AWS or GCP at v1, with explicit trigger conditions for each (KMS for HSM-backed envelope encryption, Aurora for Neon ceiling, SES at 250K emails/mo, BigQuery at \$1K Tinybird, Fargate only if Cloudflare Containers can't fit, full re-eval if HIPAA/FedRAMP ICP). §2.5 restructured into compute/edge / data-plane / vendor / app-layer groupings reflecting where each piece runs. §2.6 P0: Workers-build CI gate (wrangler deploy --dry-run on every PR) added as day-1 must-get. Library-selection rule documented. §2.6 P3: Worker → AI Gateway → upstream provider pattern; AI Gateway adds caching + provider fallback + real-time logs for free. §2.14 new "Cloud architecture": layered ASCII diagram, read-path and write-path traces, AWS/GCP trigger table, Workers constraints codified as architectural rules. V1 cost projection: ~\$15-60/mo fixed.
§1.8 decision-history entry: four tiers with clean triggers. Day-1 (zero cost) wires Sentry + structured logs + /v1/health Cron Worker because retrofitting these is painful. Tier 1 adds Instatus + billing- correctness reconciler alerts + PostHog at ~$20/mo. Tier 2 (~10 customers) adds Better Stack + Honeycomb at ~$45/mo. Tier 3 (enterprise SLA) adds Datadog/Grafana + PagerDuty + Vanta. §2.6 P0 day-1 must-gets gain Sentry, structured-logging discipline, and the /v1/health Cron Worker. §2.6 P5 explicitly routes billing- correctness reconciler alerts to Sentry+Slack and emits model_cost_usd as a UsageEvent dimension. §2.6 P6 adds PostHog free tier and ties status page to the /v1/health Cron output. §2.7 day-1 must-gets gain a new "Observability (Tier 0)" subsection. §2.15 new "Observability & monitoring" — operational state per tier, with the three platform-specific concerns codified (billing-correctness as #1 incident class; BYOK cost tracking; /v1/health as SLA evidence) and explicit "what NOT to do" cuts.
… additions
§1.9 master entry capturing six prior decisions overridden + Workers
correctness fixes + product/strategy additions from the second
adversarial panel review:
Six decision overrides:
1. Authoritative bucket_state: Upstash Redis → Durable Objects with
TTL'd reservations swept by alarms (Engineer high conviction —
kills reservation leak + Upstash availability cliff in one change)
2. P-1 gate: 1-week checklist → 3-week hard gate with dual cohort
(8 BYOK + 8 managed) and ≥3 written LOI kill criterion (PM + CEO)
3. §1.4 hedge → commit to ONE ICP after P-1 dual-cohort test
4. Trial pool: $5 any-model → $20 cheap-cached models, identity-gated
(PM + CEO; solves "too small for activation" + "too unprotected
for abuse" simultaneously)
5. SOC2 evidence collection: conditional on ICP → unconditional in P0
via Vanta/Drata (CEO asymmetric-cost argument)
6. HMAC pepper: unversioned → {pepper_version, hmac} from day 1
(Engineer; prevents rotation = re-HMAC-everything migration)
Workers footgun fixes propagated through §2:
- Hyperdrive query caching disabled for api_keys/members
- UsageEvent + AuditLog via Cloudflare Queue (not fire-and-forget,
not Postgres outbox + Cron)
- Refuse with 402/429 + X-Remaining-Budget; no silent max_tokens clamp
- Idempotency-Key state machine (in_flight | succeeded | failed)
- SENSITIVE_SCOPES bypass 60s membership cache
- Tinybird PII defense-in-depth (producer + materialized view)
- UptimeRobot external probe at Tier 0
- Typed AccountScope repo (RLS rejected — Hyperdrive bypasses)
- Free-tier daily-cap key sharding (avoid 00:00 UTC hot-key)
- Stripe Meter period-boundary handling (5-min grace, period_anchor,
+10-min reconciler, manual replay before first paid customer)
Product additions:
- P3.5 eval surface (the moat — without it v1 has no answer to "why
not LangSmith?")
- Eval/GEPA compute on Cloudflare Containers or Fly.io (does not fit
Workers 30s CPU cap)
- BYOK key spec including Mode B async/SDK-side (Noetic never sees
upstream key) — removes largest trust objection
- Success screen: 5-line @noetic-tools/sdk snippet + trace + eval
score (not curl)
- "Switching from LangSmith" as primary migration page
- §2.16 Competitive positioning section
Strategy additions:
- EU AI Act specialist call before P0
- Customer support + on-call as P0 line item
- AI Gateway behind forwardToUpstream() abstraction with exit trigger
- Audit log retention SLA 7 years (Axiom 90d hot + R2 archive)
- Cloudflare exit playbook (3-5 engineer-weeks to leave if ever needed)
- Schedule rebaseline: 12 weeks 1-engineer OR 8 weeks 2-engineer; P6
moves out of paid-beta into "first paying customer + 30 days"
Open questions resolved: free-tier shape (1K runs/500 evals/30d),
trial pool ($20 cheap-cached + identity gate). §3 specs deferred:
S1+S2 drafted in P0; S3–S7 wait for P-1 outcome; collapsed to S6
combined activation/pricing/trust.
§1.10 decision-history entry and §2.17 new operational section capturing the monetization commitment: two product surfaces (API primary, paid CLI secondary), three commercial models coexisting day 1 (subscription/seat tiers + PAYG + enterprise). Introductory pricing (first 6 months / 100 customers): - Developer $10/mo (1 seat) - Team $30/mo base + $10/seat - PAYG: TBD by P-1 - Enterprise: quote Steady-state pricing: - Developer $19/mo - Team $49/mo base + $15/seat - PAYG $0.008/agent-run + $0.03/eval-score - Enterprise typically $500+/mo Grandfather rule: existing customers grandfather at intro pricing for 12 months from signup, then migrate with 60-day notice. Unit economics: ~89% gross margin at steady-state, ~80% at introductory — BYOK eliminates inference cost, so the most expensive comp-set variable is off our P&L. Even introductory is in the black at unit level. Metering anchors: agent-runs + eval-scores. Tokens NOT billable (preserves §1.4 BYOK posture). CLI Pro feature cut: GEPA, remote evals, sub-harness commands, multi-account, plugins all paid; OSS framework remains free. API keys gain entitlements claim cached in Upstash alongside scopes; CLI offline-grace 7 days with signed local cache. Propagated through: §2.11 (Pricing-units open question resolved), §2.16 (competitive frame anchored on actual numbers), §2.6 P2 (CLI auth + entitlement check), §2.6 P5 (Stripe schema for all three models, +1 week), §2.6 P0.5 (Free-tier signup + upgrade CTA), §3 S3 (entitlements dimension), §3 S5 (no longer collapses to seat-counting).
§1.11 master entry capturing all twelve mechanical codifications.
No residual fatal flaws — three reviewers confirmed plan maturity.
Convergent (multi-reviewer):
1. Tier-change → SENSITIVE_SCOPES; two-tier CLI offline grace
(7d "no network" / 24h "tier-change pending"). Closes the
7-day license-leak window on tier change/removal.
2. PAYG vs Team anti-arbitrage: move org GEPA budget pool + shared
eval datasets + SSO from Enterprise to Team tier. Removes
"accepted arbitrage" framing — aligns Team value to eval moat.
3. §2.17 Cross-model transitions subsection with explicit dollars-
and-cents rules per direction. 7-day upgrade-then-downgrade
cooldown (downgrade immediate, upgrade prorated 1-day) closes
free-trial-of-Pro exploit. No-arbitrage invariant codified.
Strategic (CEO):
4. First-100 loyalty lock: customers signed during paid beta
grandfather at intro for 24 months + 25% off steady-state at
migration. Defends churn cliff + acts as real P-1 LOI sweetener.
5. Schedule rebaselined to 16wk-1eng (ambition 12) OR 10-12wk-2eng.
6. GEPA compute cost added to unit economics; introduces gepa-runs
as a third meter with per-tier bundled allowance.
7. agent-run definition pinned in S2 (one top-level harness.run();
sub-harness/retries/forks roll up; client retries emit new run).
8. Enterprise founding-sales motion in P-1 (design-partner template,
security questionnaire, "enterprise ready" definition).
Engineer-specialty:
9. Tier-keyed DO sharding (Free/Developer=1, Team=4, Enterprise=16).
Per-reservation settle_journal retained. Reconciler drift-
direction matrix: replay-from-journal, NEVER decrement DO.used.
DO settle/Queue-enqueue atomic via single DO method.
10. Tinybird dedup window ≥7 days as hard constraint (must exceed
max Queue retry + 5min grace + reconciler cadence). Reconciler
groups by period_anchor not ts.
11. Mode B metering-integrity contract: SDK-owned idempotency +
signed JWT with upstream request/response hash + sampled
re-query verification. Mode B gated to subscription tiers only
(PAYG requires Mode A — SDK-reported usage cannot directly bill).
Product (PM):
12. /pricing-preview page in P-1 with "reserve early-access" CTA
(not buy-now). LOIs captured as feature-checklist commitments
with prospect signature. Same URL transitions to live Stripe-
integrated flow in P5 (no SEO break).
Second-order codifications: late-settle-after-rollover policy,
Upstash-degraded fallback contract, archived audit retrieval surface
in P6, Sentry quota sampling for DO alarms + Queue DLQ, recalibrated
PAYG reconciliation tolerance, Free→PAYG cheaper than Free→Developer
funnel risk noted, P3.5 zero-slack acknowledgement.
Reaffirmed by panel: no fatal flaws; DO + Stripe Meter + period_anchor
mechanics correct; SOC2 unconditional; Idempotency-Key + SENSITIVE_
SCOPES + Hyperdrive policy accepted; LangSmith-switching wedge
uncontested.
Operationalizes the §3 spec proposal into an execution runbook gated on P-1: repository/package integration (new @noetic/api, @noetic-tools/sdk, packages/dashboard + required .sentrux/rules.toml edits), spec generation order/freeze policy with authoritative specs/saas/ file list, five-tier service provisioning runbook, secrets matrix, integrated critical-path table (provision ⊕ spec ⊕ build ⊕ milestone per phase), and cross-cutting integration checkpoints. Renumbers Next concrete step → §5, Changelog → §6.
The read/write-path narratives and layered diagram predated the §1.9/§1.11 Durable Objects override. Read path: bucket reserve/settle now via BucketStateDO (tier-sharded, TTL'd reservations, 402/429 hard-cap, settle atomic with Queue enqueue, ≥7-day Tinybird dedup); UsageEvent via Queue consumer. Write path: SENSITIVE_SCOPES strict read, no outbox table, audit via Queue → Axiom. Diagram gains a BucketStateDO box. Resolves the §4.7 doc-hygiene flag.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
plans/saas-platform.md— a living planning doc for the Noetic SaaS platform. Substantially expanded since first draft via three adversarial panel reviews + ongoing decision-locking, and now carries a concrete integration/execution runbook. Not yet at implementation stage; this is the audit-trail/RFC source we'll iterate on before P-1 (customer discovery) and P0 (foundation) start.Structure
Decisions locked
/v1/health+ UptimeRobot at day 1; Instatus + PostHog at Tier 1; defer Datadog/PagerDuty to enterprise tierSENSITIVE_SCOPES+ two-tier CLI offline grace; Team gains org-GEPA-budget + shared eval datasets + SSO (closes PAYG arbitrage); cross-model transition rules + 7-day cooldown; first-100 loyalty lock (24mo intro + 25% off steady-state); schedule rebaselined to 16wk/1eng (ambition 12) or 10–12wk/2eng;gepa-runsas a third meter;agent-rundefinition pinned; Enterprise founding-sales motion in P-1; tier-keyed DO sharding + per-reservationsettle_journal+ reconciler drift-direction matrix + atomic settle/enqueue; Tinybird dedup window ≥7 days; Mode B metering-integrity contract, gated to subscription tiers only;/pricing-previewin P-1 transitioning to live Stripe at the same URLArchitectural primitives worth scrutinizing
UsageEvent(Tinybird) withperiod_anchor, emitted via Cloudflare Queues (not fire-and-forget);(account_id, idempotency_key)dedup over a ≥7-day windowBucketStateDOper(account, metric, period, shard?)for authoritative bucket state — tier-keyed sharding (Free/Developer=1, Team=4, Enterprise=16), TTL'd reservations swept by alarms, per-reservationsettle_journal,settle()atomic with the Queue enqueue, hard-cap → 402/429 (no silent clamp)DO.used > Tinybird.sum, replay fromsettle_journal— never decrement the DO; one loop in v1, grouped byperiod_anchorperiod_anchorclamping + +10-min reconciler + manual replay tool before first paid customerAccountScoperepo for multi-tenancy (Postgres RLS rejected — Hyperdrive prepared-statement cache bypasses it); Hyperdrive query caching disabled forapi_keys/members/authz tables{prefix, pepper_version, hmac}with versioned Workers Secrets pepperSENSITIVE_SCOPES(incl.billing:tier_change) bypass the 60s membership cache; tier change fast-rotates the CLI within a 24h hard capIntegration plan (§4 — new)
Operationalizes the §3 spec proposal into an execution runbook gated on P-1:
@noetic/api(Workers/Hono, six-layer internal structure),@noetic-tools/sdk,packages/dashboard; each requires.sentrux/rules.tomllayer/boundary edits keeping the framework isolated from the SaaS appspecs/saas/NN-*.mdfile list (extends §3.5 by giving the eval surface S4.5 its own file) with drafting/freeze orderS1→S2 (lock) → (S3‖S5) → S4‖S4.5 → S6‖S7and sync-doc registrationP-1 gate → S1+S2 lock → P0/v1/healthsynthetic,wrangler --dry-runCI, cross-tenant pen-test, reconciler drift, shadow-billing)Cost-coverage invariants (§2.13)
Constraints that bound free-user marginal cost to fractions of a cent/month: (1) Redis 60s verification cache; (2) per-key RPS token bucket; (3) free-tier monthly caps with daily-cap key sharding; (4) BYOK default eliminates inference cost; (5) $20 identity-gated cheap-cached trial pool; (6) hot-path state split by consistency requirement — tier-sharded Durable Objects for atomic counters, one Upstash instance for eventually-consistent caches.
Cloud architecture (§2.14)
~95% Cloudflare; Neon (Postgres), Upstash (Redis), Axiom (audit) fill the data-plane gaps. No AWS/GCP at v1 with explicit triggers documented. Read/write-path walkthroughs and the layered diagram are refreshed to the current
BucketStateDO+ audit-via-Queue model. Cloudflare exit playbook: 3–5 engineer-weeks to leave if ever needed; abstraction points (forwardToUpstream,AccountScoperepo, DO interface, Queue producer/consumer pairs) already in place.What's NOT in this plan (§2.12)
Reselling tokens with a markup; merchant-of-record for inference; Postgres RLS; silent
max_tokensclamp; Postgres outbox + Cron audit shipper (direct Queue enqueue instead); three reconciler loops from day 1 (start with one); decrementing the DO during reconciliation (replay from journal instead); Mode B for PAYG (subscription-only); "Migrating from OpenRouter/OpenAI" as primary migration page; the §1.4 BYOK-vs-managed hedge framing; parallel P0 before the P-1 gate; whitelabel model-ID aliasing; legacy Stripe metered billing; Datadog/PagerDuty before enterprise tier.Next step (§5)
P-1 hard gate, 3 weeks, no parallel P0. Pre-call artifacts (
/pricing-previewwith locked tier shape + "reserve early-access" CTA, "Switching from LangSmith" landing, competitive teardown, EU AI Act specialist call, Enterprise founding-sales artifacts), dual-cohort discovery (8 developer/BYOK + 8 non-developer/managed), kill criteria (≥3 written LOIs at stated price + ≥8/15 trust validations). Vanta/Drata + SOC2 evidence collection kick off in parallel as non-gating items.Test plan
@noetic/api/@noetic-tools/sdk/packages/dashboardpackages + their sentrux boundariesspecs/saas/location + spec freeze policy (lock S1+S2 before drafting S3–S7?)