Skip to content

Commit 034e207

Browse files
AbirAbbasclaude
andauthored
feat: agent-native SaaS platform (PRD #184, R1–R8) (#187)
* feat(lifecycle): migration 00035 — invitations, billing-event idempotency, service-account keys, billing role Adds the R8 lifecycle schema on top of existing primitives: - suite_api_keys.service_account_name (labels non-human keys) - 'billing' role widened into suite_memberships CHECK - suite_invitations (token-based, FORCE RLS) with accept/revoke/expire states - suite_billing_events idempotency ledger (unique tenant_id,event_id, FORCE RLS) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(lifecycle): tenant-role capability matrix (owner/admin/member/billing/viewer) Pure authorization matrix for a caller's role on their OWN tenant, separate from internal/rbac (operator/Casbin). Gates the tenant-management surface: viewer/member cannot mint keys, billing role gets billing only, tenant.manage (delete/transfer) is owner-only. Fully unit-tested, no I/O. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(lifecycle): invitations package — token state machine + suite_invitations store Pure accept/revoke/expire state machine (CanAccept/CanRevoke/EffectiveStatus) plus sha256 token hashing (plaintext shown once, only the hash persists) and a Postgres store. Accept looks the row up by token_hash under bypass_rls (the invitee has no tenant binding yet) and enforces an email match so a leaked token can't be redeemed by another account. State machine fully unit-tested. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(config): add AF_STACK_ENV + sandbox network policy for the prod contract R7 production operating contract foundation. Adds a Config.Env field (AF_STACK_ENV) and the ProductionHardening() gate (saas mode AND env=production) that the startup preflight and /ready posture checks key off. Adds Sandbox.NetworkPolicy (AF_STACK_SANDBOX_NETWORK_POLICY, default "isolated") so the preflight can refuse a production deployment that ships sandboxes on the open bridge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(secrets): tenant-scoped secrets vault surface at /api/v1/vault/secrets The secrets vault only exposed an operator-gated surface (/api/v1/secrets, hardcoded default tenant), so SDK calls over a tenant bearer key 403'd even though both SDK clients document per-tenant secret operations. Add a separate tenant-principal surface at /api/v1/vault/secrets that rides the tenant resolver (the prefix is deliberately NOT in publicPrefixes, so tenantResolver binds the caller's tenant before any handler runs). Every operation scopes to that tenant via the vault's WHERE tenant_id clause and the existing FORCE ROW LEVEL SECURITY policy on suite_secrets. No schema change is needed: suite_secrets already has tenant_id + RLS (00003/00004). - list / get-metadata return metadata plus a 'secret:<key>' reference, never plaintext; plaintext leaves only via the explicit /reveal contract. - set / delete / rotate / reveal are audit-logged via the existing audit system, anchored at the resolved tenant. - the operator surface in secrets.go is untouched (separate audited surface). - a tenant-less context is refused (401) rather than falling back to the default tenant (defence-in-depth). Tests inject a fake secrets.Store (via a test-only s.secretStore override) and assert the set/list/get/delete round-trip, no-plaintext-on-read, cross- tenant isolation, rotate semantics, the 503 no-vault path, and that the operator surface stays gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(prodcheck): production posture checks (RLS, CORS, KMS, storage, sandbox) New internal/prodcheck package: pure, unit-tested check functions plus thin pgx gatherers. Verifies the serving DB role can't bypass RLS, every tenant-owned table (derived dynamically from a tenant_id column) has RLS enabled AND forced, no wildcard credentialed CORS, a real (non-dev) KMS key, object storage tenant-isolated when configured, and sandbox network policy isolated. Each check carries a stable code for the boot log / readiness envelope. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(sdk-py): point secrets client at the tenant vault surface The Python secrets client documented per-tenant operations but targeted /api/v1/secrets, which the runtime gates operator-only — so a tenant bearer key 403'd. Repoint every verb (get/reveal/list/put/delete/rotate) at the new tenant-principal surface /api/v1/vault/secrets/*, authorized by the caller's own API key. Add the optional 'reference' field (secret:<key>) returned by tenant-surface reads and update the mocked tests to the new paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(server): fold production posture into /ready When the production contract is armed (saas + AF_STACK_ENV=production), /ready re-evaluates the prodcheck posture so drift after boot (e.g. an operator ALTERs a tenant table's RLS off) turns the pod un-ready with a stable prodcheck_code in the envelope. Cached with a 30s TTL so readiness probes never hammer pg_catalog. No-op outside production. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(sdk-ts): point secrets.get at the tenant vault surface secrets.get targeted /api/v1/secrets/{key}/reveal, which the runtime gates operator-only — a tenant bearer key 403'd. Repoint it at the tenant- principal surface /api/v1/vault/secrets/{key}/reveal, authorized by the caller's own API key, and update the mocked test URLs. Kept to the single existing verb; SDK-wide parity is owned separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(runtime): production preflight refuses to boot a mis-hardened deployment Adds runProductionPreflight (cmd/af-stack): when saas + AF_STACK_ENV=production, it gathers the live catalog + config posture and os.Exit(1)s with a stable code per failing check (RLS-unsafe role, un-forced tenant tables, dev KMS key, wildcard credentialed CORS, un-isolated storage, open sandbox network). There is deliberately no bypass flag — a production deployment that can't prove isolation must not accept traffic. One call site in main.go before the server is built. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(billing): pure subscription reconciliation decision logic Reconcile()/EntitlementPlan() map a Stripe subscription snapshot onto the local plan+status the runtime should enforce: active/trialing/past_due keep the subscribed plan (past_due = dunning grace), terminal states (canceled/unpaid/incomplete_expired/paused/incomplete) degrade entitlements to the default plan. No I/O — every drift case is unit-tested with fabricated payloads. Shared by the webhook handler and the reconcile cron so they can't diverge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(billing): dunning, entitlement degrade, idempotent ingestion + reconcile cron - webhook: invoice.payment_failed -> past_due (dunning, entitlements retained); subscription.updated degrades to default plan on terminal status; every ingest path guards on suite_billing_events for idempotency (Stripe retries). - store: UpsertCustomer now preserves the plan when the caller passes an empty plan (fixes a latent bug where invoice.payment_succeeded downgraded a paid tenant to free); adds RecordBillingEvent idempotency helper. - ReconcileSubscriptions cron re-syncs live Stripe state to correct drift from missed/out-of-order webhooks, via a Stripe-only subscriptionReader that no-ops for stub/lago/remote adapters. Reuses the pure Reconcile decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(idempotency): add suite_idempotency_keys migration (R1) Tenant-owned idempotency store table with tenant_id + FORCE ROW LEVEL SECURITY per the 00004/00032 pattern. Records, per (tenant_id, idempotency_key), the request fingerprint and captured response for replay. status_code NULL marks an in-flight reservation; non-NULL marks a completed, replayable row. expires_at defaults to 24h; a unique index on (tenant_id, idempotency_key) backs atomic ON CONFLICT claims and an expires_at index backs opportunistic purges. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(idempotency): store package with reserve/replay + fingerprint (R1) New internal/idempotency package: the Store interface (Reserve/Save/ Release) plus a Postgres-backed PgStore and a stable request Fingerprint helper (method+path+query+body hash). Reserve atomically claims a key via INSERT ... ON CONFLICT DO NOTHING and, on conflict, classifies the existing row as replay / mismatch / in-flight. All methods scope to the tenant on the context, relying on the pool's PrepareConn RLS binding. The exported Store interface is the small helper other subsystems (webhooks/billing) can consult to make their own inbound operations idempotent. Unit tests cover fingerprint determinism and every Reserve outcome via a fake pgx pool (no DB required). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(idempotency): inbound middleware hook + response capture (R1) Wire the idempotency store into the runtime handler chain: a single middleware that sits inside the tenant resolver (so the tenant is bound) and wraps every handler. It acts only on mutating (POST/PUT/PATCH/DELETE) /api/v1 requests carrying an Idempotency-Key with a resolved tenant. Behavior: a retry with the same key + fingerprint replays the stored response byte-identically with Idempotency-Replayed: true and does not re-run the handler; a reused key with a different fingerprint returns 422 IDEMPOTENCY_KEY_REUSED; a concurrent duplicate returns 409 IDEMPOTENCY_IN_FLIGHT. GET/HEAD, missing header, no tenant, non-/api/v1, and oversized request bodies pass through. Streaming (SSE), >1MiB, and 5xx responses are not stored (the reservation is released). Bookkeeping errors fail open. Response capture is added to the existing statusWriter as an optional tee (write-through, so streaming and request_id stamping are unaffected) with a 1MiB cap. Store construction is nil-safe: absent a DB pool the middleware is not installed. Wired httptest coverage drives the real middleware stack against a fake store. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(migrationlint): static safety lint for goose migrations New services/runtime/cmd/migrationlint tool + scripts/migration-safety-check.sh wrapper + make lint-migrations target. Flags migrations missing an Up/Down section, DROP TABLE/COLUMN in the Up direction without a `-- backai:allow-destructive` marker (Down-section rollback DROPs are ignored), and unbalanced goose StatementBegin/StatementEnd pairs — the DO-block hazard that only surfaces at apply time. Unit-tested against seeded bad fixtures plus a regression guard proving the shipped 32 migrations lint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(lifecycle): tenant delete revokes keys, purge cron, ownership transfer, billing role - DeleteTenant now soft-deletes AND revokes all the tenant's active API keys atomically, so credentials stop authenticating the moment deletion lands (data retained for the grace period). - PurgeSoftDeletedTenants hard-deletes tenants past an N-day grace window (FK cascade tears down members/keys/invitations/billing); rejects graceDays<=0. - TransferOwnership promotes a member to owner and demotes prior owner(s) to admin in one tx (never ownerless, never two owners). - MembershipRole resolves a caller's tenant role for capability gating. - 'billing' added to the valid membership role set. Integration tests (tag: integration) cover each; excluded from default build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * refactor(cost): extract pure budget deciders + prove multi-replica correctness Audit outcome: budget/per-key enforcement holds no in-memory spend state — every decision reads committed spend from the shared suite_cost_events ledger and records back to it, so two replicas enforce against identical state. Extract the decision into pure budgetDecision/keyBudgetDecision (enforce.go) documenting the shared-state property, and add a non-integration two-enforcer test: a call recorded by replica A is seen by replica B, concurrent enforcers across replicas never breach the cap, and the admitted total is exact. Runs clean under -race. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(lifecycle): resolver surfaces 401 KEY_EXPIRED for expired API keys Expired keys now return a distinct KEY_EXPIRED envelope instead of the generic INVALID_API_KEY, so a caller who legitimately holds an expired key can tell it apart and rotate. Safe: VerifyKey checks expiry only AFTER bcrypt succeeds, so this never leaks whether an arbitrary key exists; all other failures still collapse to INVALID_API_KEY. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(openapi): declare per-operation required scope + principals Add x-required-scope / x-principals extension fields to the Operation type and an AnnotateSecurityFunc walker so a central scope registry can attach security metadata to every registered operation without editing each Register call site. Purely additive — no existing route changes until a caller wires the annotation pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(auth): enforce per-route API-key scopes + RLS-safety readiness gate Central route->required-scope registry (scopes.go) consulted by the tenant resolver for every authenticated API-key request. A key with empty/["*"]/"admin" scopes keeps full tenant access (legacy default — the 60-second quickstart is unaffected); a key with narrow scopes is fail-closed with 403 SCOPE_DENIED naming the missing scope. Session and default-tenant principals are never scope-gated. The same registry drives the OpenAPI x-required-scope / x-principals annotations so the spec can't drift from enforcement. resolveBearer now returns the full key (scopes included) behind a keyVerifier seam so scope enforcement is testable without Postgres, and the best-effort key-touch is guarded on a live tenancy manager. Also adds the /ready "rls_unsafe" gate: when the runtime intends to isolate tenants (SaaS + multi-tenancy) but the serving DB role can bypass row-level security, readiness reports 503 so a load balancer never routes tenant traffic to a pod that cannot isolate it (flag computed at boot). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(boot): mode-aware fail-closed SaaS RLS boot check Make the tenant-isolation boot gate mode-aware: effectiveMTEnabled folds personal mode into the predicate so the resolver's multiTenancyEnabled() and the boot check can never disagree. In SaaS mode with multi-tenancy on, an RLS-bypassing serving role stays fatal (unless AF_STACK_ALLOW_INSECURE_DB overrides); personal mode is relaxed. assertTenantSafeRole now returns whether the deployment booted in the unsafe-but-overridden state and hands it to server.Deps.RLSUnsafe so /ready can report it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(sdk-py): transport class with retry + version policy Refactor af_stack/_http.py around an explicit Transport (base_url, api_key, timeout, max_retries) plus a contextvars-bound active transport, so both the env-configured default path and explicit clients share one code path. - Retry policy: 429/5xx retried with exponential backoff honoring Retry-After; only GET/HEAD/OPTIONS (or calls carrying an idempotency_key) retry. - Idempotency-Key header threaded through request_json. - Lazy runtime-version probe (GET /version, tolerates 404) + check_runtime_compat pure helper + SUPPORTED_RUNTIME_RANGE constant. - AFStackError gains a .status alias (parity with TS SuiteError). - Env-default transport keeps legacy behaviour (no retries); module-level request_json/stream_sse/get_client/_build_headers now delegate to it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(sdk-py): auto-iterating pagination helper Add af_stack.paginate / AsyncPaginator: a transport-agnostic async-iterable that walks either offset-based (limit/offset + has_more) or cursor-based (next_token) page fetchers to exhaustion, with .pages() and .collect(limit=). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(metrics): add operating-contract metrics for the default alert rules Adds backai_budget_rejections_total (wired from the gateway 402 path, by tenant+reason), backai_jobs_queue_oldest_age_seconds and backai_db_pool_{acquired,max}_connections (refreshed by a 30s background sampler that reads pool stats + the oldest past-due river_job), and backai_backup_test_last_success_timestamp (set by the backup-test cron). Every series is zero-initialised so alerts have a metric to evaluate from boot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(sdk-py): explicit BackAI client + parity manifest Add af_stack.BackAI — an explicit, configurable client exposing the governed operational namespaces (agents, jobs, storage, llm, memory, search, webhooks, billing, ...). Each method is bound to the client's own Transport via a contextvars slot, so existing namespace modules target the client's config with no per-module rewiring. Env-configured suite.* singletons are unchanged. - packages/sdk-parity.json: cross-language parity contract (canonical/py/ts names per governed namespace). test_parity.py asserts the client surface matches it exactly, so a method added in one language only fails tests. - idempotency_key option threaded onto agents.call/call_async + jobs.enqueue (sends Idempotency-Key header; enables safe retry of the mutation). - Lazy runtime-version check wired into every explicit-client call (once). - Exports: BackAI, AFStackError, Transport, paginate, AsyncPaginator, check_runtime_compat, SUPPORTED_RUNTIME. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(lifecycle): thread service_account_name through key issue/list/rotate Adds ServiceAccountName to APIKey + IssueAPIKeyInput and persists/reads suite_api_keys.service_account_name across IssueKey, issueLocalKeyWith, listKeysImpl, and RotateKey (rotation preserves the label). Lets a key be minted as a named non-human service-account credential. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(jobs): pull-based remote worker protocol + DB-backed rendezvous Remote (python/typescript) job definitions were rejected at Enqueue and cancelled by the executor — there was no cross-language dispatcher. Add a PULL-based remote worker protocol so those kinds run out-of-process while River keeps the durable queue + retry semantics. Core (internal/jobs): - suite_job_leases / suite_job_worker_logs migration (00033): the DB-backed rendezvous, both tenant-owned with FORCE ROW LEVEL SECURITY. - RemoteStore contract (remote_store.go) with a faithful in-memory impl for tests and a Postgres impl (tenant predicate + RLS, FOR UPDATE SKIP LOCKED lease pop, idempotent enqueue that supersedes older attempts). - Executor (remote_executor.go): the umbrella worker registers a `ready` attempt and blocks polling it. Outcome -> River return: completed=nil, retryable-fail=error (retry/backoff/dead-letter), permanent-fail=JobCancel, lease-TTL-expiry=retryable error, unleased=JobSnooze (free the slot), cancellation propagates via the job context + a canceled flag the worker sees on its next heartbeat. - Enqueue now accepts python/typescript the same as go; the umbrella worker hands remote kinds to the executor instead of cancelling. Tests (no DB): lease/heartbeat/complete/fail state machine, tenant-scoped + kind-scoped leasing, cross-tenant denial, TTL-expiry, idempotency 409, and the executor's pure evaluator + River-outcome mapping + await loop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(server): REST surface for the remote job worker protocol Add POST /api/v1/jobs/worker/{lease,heartbeat,complete,fail,logs} so out-of-process python/ts workers can drive remote job execution. - Auth: a tenant API key carrying the jobs:work scope (personal mode uses the default tenant, no scope gate). Leasing is tenant-scoped — a key only ever sees its own tenant's attempts (enforced in-handler AND by FORCE RLS in the store). - lease is a long-poll (up to ~25s) that returns the oldest ready remote attempt among the kinds the worker declares, or {job:null} on timeout. - heartbeat extends the lease TTL and reports cancellation; complete/fail resolve the attempt (fail carries a retryable flag); logs attaches structured lines to the run. - Store sentinels map to the structured envelope: 404 LEASE_NOT_FOUND, 409 LEASE_CONFLICT / LEASE_ALREADY_RESOLVED, 503 JOBS_NOT_CONFIGURED. server.go gains a single registerJobsWorkerRoutes() call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(modules): declarative workload-module engine (discovery, RLS-lint, CRUD) PRD R2: turn the empty internal/modules Registry into a real filesystem-discovered, declarative workload-module system. - Manifest (backai.module.yaml): id/name/version/description/resources with typed fields (string/int/bool/timestamp/json, required, defaults), enabled-by default false. Strict parse + validation (reserved columns, duplicate resources/fields, semver, default-type coercion). - SQL RLS-lint: statically refuses any module migration whose CREATE TABLE lacks tenant_id + ENABLE/FORCE ROW LEVEL SECURITY + a policy. A tenantless table disables the module (structured error) while the runtime keeps serving everything else. - Declarative CRUD: auto-generated tenant-scoped list/create/get/update/ delete. The query builders ALWAYS filter tenant_id = $1 (bound from the resolver context, never the client body) — cross-tenant access is impossible by construction, with Postgres RLS as the backstop. - Migration applier: versioned .sql under <module>/migrations, tracked in the platform registry table; per-file transactions. - Manager: discovery + classification (active/inactive/errored), route mounting, and the operator inventory (id/version/health/migration state). - Resource->table convention: <module>_<resource> (documented). Tests cover manifest parse/validate, the RLS-lint (accepts notes-shape, rejects tenantless + each missing isolation statement), builders always filter tenant, handlers bind the resolver tenant and ignore body-supplied tenant_id, and route generation (disabled module => no routes). DB-bound apply paths are t.Skip (verified live downstream). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(db): platform registry tracking applied workload-module migrations Migration 00036 adds suite_module_migrations (module_id, version, name, applied_at), keyed by (module_id, version). Platform-owned bookkeeping — intentionally no tenant_id/RLS, like suite_tenants and the goose version table; a module's own data tables are the tenant-scoped, RLS-forced surface. Lets each workload module advance its schema idempotently across reboots. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(config): WORKLOAD_MODULES_PATH for declarative module discovery Add Modules.WorkloadModulesPath (env WORKLOAD_MODULES_PATH, yaml workload_modules_path), default ./workload-modules — the filesystem root the runtime scans for <id>/backai.module.yaml. The enabled set continues to come from the existing Modules.WorkloadModules list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(server): mount workload-module CRUD + operator GET /api/v1/admin/modules Minimal server-side wiring for the declarative module system: a new Modules dependency (nil-tolerant) and a single registerWorkloadModuleRoutes call. It hands the manager the runtime pool + canonical writeJSON/writeError (so module responses carry request_id), mounts every routable module's tenant-scoped CRUD under /api/v1/workload/<module>/<resource>, and exposes the operator inventory at GET /api/v1/admin/modules (gated via the existing admin:* Casbin policy). Disabling a module removes its routes on next boot without dropping data. main.go wiring (Load + ApplyMigrations before New) is downstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * docs(modules): notes reference workload module A committed, living example: backai.module.yaml declaring a notes resource (title/body/done) plus an RLS-compliant migration creating notes_notes with tenant_id + FORCE RLS + a tenant-isolation policy. enabled: false so it ships as an example without auto-serving. README documents the resource->table convention, the tenant-isolation contract, and how to enable it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): scaffold the declarative module shape in `af-stack module new` Replace the dead-end scaffold (manifest.yaml with a routes: block + handlers/routes.go.example) with the real PRD R2 shape: a backai.module.yaml declaring an items resource and an RLS-compliant migration (tenant_id + enable/force RLS + policy) that passes the runtime's static lint, plus a README. Update the scaffold test to assert the new manifest + migration content. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(alerts): default Prometheus alert rules for the operating contract deploy/prometheus/alerts.yml covers readiness flapping, LLM provider failure rate, jobs queue backlog, budget-rejection spikes, webhook delivery failures, DB pool saturation, and backup-test staleness. Adds a backai_webhook_deliveries_total counter (wired at the outbound delivery chokepoint) so the webhook alert has a real signal. A Go test parses the rule file, asserts required coverage, and — crucially — fails if any rule references a backai_* metric the runtime does not register. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(connections): add suite_connections + events schema (R5) Tenant-owned connection contract for external services. Both tables get tenant_id + FORCE ROW LEVEL SECURITY (per 00032) so isolation is enforced at the database. encrypted_credentials reuses the AES-256-GCM envelope that backs suite_secrets; webhook_secret_ref points at a vault entry rather than storing a signing secret inline. connection_events is an append-only audit trail (created/refreshed/revoked/health_check/error). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(connections): connections subsystem (providers, crypto, handle, webhooks) New internal/connections package implementing R5's secure connection contract: - Typed provider capability descriptors in an in-code registry (github, stripe, google, slack). Adding a provider = adding a Descriptor, not expanding trusted core logic. - Credentials sealed with the shared secrets cipher (same envelope as the vault); the plaintext credential is never returned or logged. - Service.Handle is the credential-injecting proxy: it loads the connection, single-flight-refreshes an expired OAuth token, enforces granted scopes where the provider exposes a mapping, injects auth server-side, refuses absolute paths (credential-retargeting guard), and relays the provider response. The auth header is redacted from logs. - Concrete Stripe + GitHub webhook HMAC verification with a scheme dispatcher. - Store interface with a Postgres-backed impl (FORCE-RLS rows, vault-backed webhook secret refs) plus an in-memory fake for tests. - OAuth authorize-URL / signed-state / callback exchange. - Lifecycle audit events emitted on every transition. Tests cover the R5 acceptance set: encrypt/decrypt roundtrip via a fake key, single-flight refresh against a fake provider (exactly one refresh under concurrent load), auth injection + log redaction, Stripe/GitHub signature vectors, and revoked-connection => ErrRevoked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(connections): wire /api/v1/connections routes into the runtime Thin server handlers over connections.Service: create (api_key stores a credential and never echoes it; oauth returns an authorize URL + state), list (metadata + health), delete (revoke), the request HANDLE, and verify-webhook, plus the browser OAuth callback. Errors map to stable codes (CONNECTION_REVOKED => 409, etc.) via the shared envelope. server.go gains one field + one registration block; the /connections/callback path is added to the tenant-resolver public prefixes (signed state carries the tenant, like /oauth/callback) — note this does NOT match /api/v1/connections/*, which stay tenant-resolved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(connections): construct the connections service at boot Build the R5 connections subsystem alongside the secrets vault (it shares the KMS-backed cipher and needs the DB pool). The outbound provider HTTP client is SSRF-guarded via safehttp; platform OAuth client creds resolve from each descriptor's env vars (CONNECTIONS_<PROVIDER>_CLIENT_ID/_SECRET). Passed into server.Deps.Connections; nil leaves /api/v1/connections at 503. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(sdk-py): Worker for language-neutral background jobs af_stack.worker.Worker(base_url, api_key).register("kind")(fn) + run() — a long-running process that pulls remote (python) job definitions from the runtime and executes them via the pull-worker protocol. - run() long-polls /jobs/worker/lease, dispatches to the registered handler, and reports the outcome; heartbeats run on a background thread so the lease never expires while a handler works. - Handlers get (payload, ctx): ctx carries tenant_id/job_id/attempt/deadline, ctx.log(msg, **fields) attaches structured lines to the run, and ctx.is_canceled() reflects heartbeat-reported cancellation. - Failure semantics: return -> complete; raise PermanentError -> permanent fail (no retry); raise anything else -> retryable fail. - Graceful drain on SIGTERM/SIGINT: stop leasing, finish the in-flight job, exit. Tests mock the runtime over respx: lease, complete-with-result, retryable vs permanent failure, ctx.log, heartbeat cancellation, and graceful drain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(lifecycle): customer-facing tenant self-management routes (/api/v1/me/*) Session-principal, role-gated (tenantrole matrix) surface for a tenant to manage itself: list/mint(expiry+scopes+service_account_name)/revoke own API keys, create/list/revoke invitations (email via notifications), accept an invitation by token, transfer ownership + soft-delete the tenant (owner only), and read the tenant audit trail. Key + invitation ops are tenant-ownership guarded. Accept bypasses the resolver (invitee has no membership yet) and self-authenticates via token + session. Minimal server.go insertion (one registration call) + one publicPrefixes entry for the accept route. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(backup): backup/restore verification script + opt-in platform cron scripts/backup-restore-test.sh dumps the source DB, restores into a throwaway scratch database, and compares row counts on core tables — exiting non-zero on any mismatch, so a green run is real evidence the backup is recoverable. It defaults the source to the privileged migrate role and warns that FORCE-RLS tenant tables dump empty under a restricted role. registerBackupTestCron wires it as a platform system cron (same scheduler as retention.daily) when BACKUP_TEST_ENABLED=true; on success it stamps backai_backup_test_last_success_timestamp so the staleness alert has a signal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(lifecycle): platform crons — tenant purge, invitation expiry, subscription reconcile Registers three system crons on a dedicated scheduler (after tenancy + billing are constructed): lifecycle.tenant_purge (daily, hard-deletes tenants past the AF_STACK_TENANT_PURGE_GRACE_DAYS window, default 30), lifecycle.invitation_expire (hourly), and billing.reconcile (every 6h, Stripe drift correction). Each guards on the presence of its dependency and only starts the scheduler when at least one registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(lifecycle): self-service session list + revoke (/api/v1/me/sessions) Lets any signed-in member list their own active better-auth sessions (token never exposed; current session flagged) and revoke one, scoped to the caller's own sessions via an email join. Gated by the self.manage capability. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * fix(correlation): stamp the request_id onto the request so cost events join back withLogging computed the canonical correlation id and set it on the response + log trace_id + error envelope, but never onto the request — so llmRequestID (the source of cost_events.request_id, migration 00026) derived a *different* id and the request → cost-event → webhook chain couldn't be joined. Stamp the same id onto r.Header["X-Request-ID"] before the handler runs. The webhook leg already propagates SendInput.Headers to the outbound POST, so an emitter that forwards X-Request-ID carries it end-to-end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): output package — stable exit codes + --json contract One helper for the machine contract every command shares: documented exit codes (0 ok, 1 error, 2 usage, 3 auth, 4 not-found, 5 validation, 6 remote), a Fault error carrying an explicit code, an ExitCode mapper (Fault > client.APIError status > flag.ErrHelp > generic), a single EmitJSON, a Result dual-renderer, and ParseArgs for flags interspersed with positionals. Table-driven exit-code test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): client Probe helper for root-level /health and /ready Do() prefixes /api/v1; readiness probes live at the root. Probe returns (status, body, err) and reports transport failure as err with status 0 so callers can tell 'runtime down' from 'responded non-2xx' and degrade gracefully. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): validate package — module manifest + migration RLS + agent Offline validators shared by test/module validate/agent validate. Manifest shape (id slug, semver, routes, meters), the multi-tenancy invariant lint (any table with tenant_id MUST enable+FORCE RLS with a policy on app.tenant_id), and agent-scaffold structure. Accepts both backai.module.yaml and the legacy manifest.yaml. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): init --template saas scaffolds a complete project New saas template emits a full tree: a lean Vite+React+TS customer app with an auth-aware /api/v1 client (login + notes page), one notes workload module (backai.module.yaml + a tenant_id+FORCE-RLS migration), a notes-assistant agent, a vitest test, .env.example, a compose reference, AGENTS.md/CLAUDE.md, and a machine-readable capabilities.json. init gains --json. Golden-file test asserts the tree and validates the generated module end-to-end; the dependency-free API client typechecks with a bare tsc (skipped when tsc is absent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): doctor, status, test — diagnostics that degrade offline doctor checks docker/node/env/ports/reachability/migrations and exits non-zero only on a critical failure (a down runtime is a warning). status is a compact up/config snapshot. test runs the fork gates (module-manifest, migration-rls, scaffold-typecheck, sdk-smoke), each pass/fail/skip; live-dependent gates skip offline. All support --json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): db diff|push|generate|reset — goose wrapper, guarded reset Thin wrapper over goose for core + module migrations. reset is destructive: requires --yes and honours --dry-run; missing goose or DATABASE_URL degrade to a clear structured error. Table-driven tests assert the argv and the destructive guard via an injected runner. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * fix(db): renumber remote_job_leases migration 00033 -> 00037 Two parallel workstreams claimed slot 00033; idempotency keeps it, the remote-worker lease table moves to the next free slot so goose applies both in order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(auth): wire cross-stream route families into the scope registry The scope registry and the worker/vault/connections/workload surfaces were built in parallel branches, so the registry predates the routes: - /api/v1/jobs/worker/* now takes the dedicated jobs:work scope; the registry's general jobs:write mapping would have 403'd a lease-only worker key at the resolver before its in-handler jobs:work check ran - /api/v1/vault/secrets/* enforces secrets:read/write (reveal + rotate are writes) - /api/v1/connections/* enforces connections:read/write - /api/v1/workload/* (generated module CRUD) enforces workload:read/write Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * fix(cli): align saas scaffold + validator with the runtime module contract The saas scaffold and the validate package were built in parallel with the workload-module loader and guessed at an imperative manifest shape (routes:/handler:/meters:) the runtime never implemented. The runtime's contract is declarative (resources: with typed fields) and its parser rejects unknown keys, so the scaffolded module would have refused to load. Worse, the scaffolded migration was a goose file, but the module runner executes migration SQL raw — the '-- +goose Down' section's DROP TABLE would have executed during apply, right after creating the table. - scaffold: declarative manifest (resource notes -> table notes_notes), plain forward-only SQL migration with the runtime-managed updated_at column, docs updated - validate: manifest schema now mirrors services/runtime/internal/modules/manifest.go rule for rule; the legacy imperative shape gets a targeted error pointing at the declarative contract The scaffold test already pipes the generated module through validate.ModuleDir, so scaffold and validator can no longer drift apart silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): af-stack job new — background-worker scaffolder (py|ts) Scaffold a pull-based background-worker job into jobs/<name>.{py,ts} from the worker SDK pattern: register the slugified kind, a handler taking (payload, ctx), and a run entrypoint. Pure local scaffold — no runtime, no key. Prints next-steps (how to run it + the jobs:work scope requirement); --json emits {"created": [paths]}. Refuses to overwrite an existing file and maps arg/flag errors to the stable exit codes. Golden fixtures pin both language templates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): af-stack connection add|list|remove Thin wrappers over /api/v1/connections using the existing CLI client + AF_STACK_URL/AF_STACK_API_KEY config. `add --kind api_key` reads the credential from a prompt or from stdin (--credential-stdin) — never from argv, which would leak into shell history — and sends it in the api_key body field; `add --kind oauth` prints the authorize URL the runtime returns. `list` shows metadata + health only, never a credential. `remove <id>` confirms interactively unless --yes. All support --json and the standard exit codes (runtime status maps 401/403->3, 404->4, 4xx-validation->5, 5xx->6). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): af-stack secrets set|list Wrap the per-tenant vault (/api/v1/vault/secrets). `set <key>` reads the value from a prompt or from stdin (--value-stdin) — never from argv — and PUTs it; the plaintext never travels as a CLI arg. `list` shows metadata (key, updated_at, reference, description) only, never a value. Both support --json and the standard exit codes; a 503 (vault not configured) surfaces as the remote exit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): logs --json/--tail/--since + remote-error degrade Extend `af-stack logs` to the standard contract: add --json (emits the {"logs": [...]} envelope via output.Result), --tail (recent-N alias for --limit), and --since (forwarded as a query param). A runtime that lacks the /admin/logs route degrades to exit 6 (remote/capability gap) instead of a bare 404, so scripts can tell "no logs surface" from "target missing". Query params are now URL-escaped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): register new + dangling commands, honor exit-code contract Wire job/connection/secrets into the root command tree, plus the already -built-but-unrouted db/doctor/status/test packages, and update the help text + examples. Switch the top-level error tail from a blanket os.Exit(1) to os.Exit(output.ExitCode(err)) so every command's failures carry their documented stable exit code — the contract the output package was built for and that the new commands rely on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * docs(cli-admin): document connection/secrets/job, logs flags, exit codes Add sections for the app-developer commands (connection, secrets, job) with their key requirements, credential-via-stdin note, and REST endpoints; document the new logs --json/--tail/--since flags and the missing-route degrade; add the stable exit-code table and correct the intro (not every command needs an operator key). Brief diagnostics & migrations table for the newly-wired status/doctor/test/db. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(modules): compose wiring for workload-module discovery The runtime image ships only the binary, so the module loader had no filesystem to scan in the compose stack. Mount ./workload-modules ro into the runtime and add an AF_STACK_WORKLOAD_MODULES env override for the enabled list (comma-separated), matching the env-first posture the other module flags already use. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(sdk-ts): explicit BackAI client, transport parity + pagination (R6) Mirror the Python SDK's R6 (SDK DX) surface in TypeScript, governed by packages/sdk-parity.json: - BackAI: explicit `new BackAI({baseUrl, apiKey, timeout?, maxRetries?})` exposing the same governed namespaces as the env-configured singleton. Config binds ambiently (the TS analogue of af_stack's contextvar transport) so namespace modules target the client with no rewiring. - Transport parity in _http.ts: retries on 429/5xx with backoff honouring Retry-After (numeric + HTTP-date, 20s cap), NEVER auto-retrying a non-idempotent mutation unless an idempotencyKey is supplied (which also sends the Idempotency-Key header); per-request timeout via AbortSignal; ambient-aware resolveBaseUrl/resolveApiKey (storage/realtime/runs routed through them). Singleton path unchanged (maxRetries 0, no timeout). - pagination.ts: AsyncPaginator + paginate (offset + cursor), 1:1 with af_stack.pagination. - Runtime-version policy: SUPPORTED_RUNTIME + checkRuntimeCompat; BackAI lazily probes GET /version once and warns on a major mismatch (tolerates 404 as unknown). - Server/browser export split: package.json exports add ./browser (excludes the privileged admin surface) alongside the default node entry. Tests: parity (TS surface == sdk-parity.json), pagination, transport (retry/Retry-After/idempotency/timeout/version), client (config binding + version warn). All 277 pre-existing tests stay green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(sdk-ts): pull-based Worker + server export split (R3) Mirror af_stack.worker and the Go worker protocol (/api/v1/jobs/worker/*): - Worker.register(kind, handler) + run() long-poll lease loop, heartbeats on an interval, complete/fail with a retryable flag, graceful SIGTERM/SIGINT drain. JobContext exposes {tenantId, jobId, attempt, deadline, log, isCanceled}. PermanentError -> dead-letter; any other throw -> retryable. - The Worker is server-only (installs process signal handlers, long-polls with a jobs:work key), so it is exported exclusively from the new ./server entry (@af-stack/sdk/server) — the default and ./browser entries never expose it. Tests: fetch-mocked worker suite (lease, complete + payload roundtrip, retryable/permanent failure, cancellation skip, heartbeat, structured log, graceful drain). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * test: live SDK + worker conformance harnesses + Makefile targets Two standalone, self-reporting harnesses meant to run downstream against a live stack (BASE_URL + API_KEY env), emitting {pass, fail, ...} JSON and exiting non-zero on failure: - scripts/sdk-conformance/run.py (af_stack) and run.ts (@af-stack/sdk) run the IDENTICAL checks so a drift in either SDK's live behaviour is caught: /health, agents list + supportdesk.echo call, storage roundtrip, jobs enqueue + status, an intentional auth-denied (401/403) and a 404, each asserting the structured error envelope. Checks unavailable on a given stack (module disabled, auth off) are SKIPPED, not failed. - scripts/worker-conformance/ exercises the pull-worker protocol end to end: spec.json vectors (success, retryable->retry->success, permanent->dead- letter, payload roundtrip fidelity, heartbeat-keeps-long-job-alive), a reference worker in each language (ref_worker.py via af_stack.worker, ref_worker.ts via @af-stack/sdk/server), and run.sh which enqueues the vectors and verifies terminal states via the jobs API. Prereq documented in-file: the vector kinds must be registered as remote job definitions. - Makefile: `make sdk-conformance` (runs both SDK harnesses) and `make worker-conformance` (WORKER=py|ts). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * chore(lint): clear golangci new-issues gate across swarm code 16 findings from the parallel workstreams: exhaustive switches now enumerate their enum members, builtin shadowing (max/cap) renamed, prealloc + unused-param + unused-func cleanups, G101 nolints on the prodcheck status-code constants (they pattern-match credential names but are stable operating-contract codes), and the Stripe take-first loop rewritten as an if (SA4004). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(jobs): declare remote job kinds via AF_STACK_REMOTE_JOB_KINDS RegisterRemote had no production caller, so a Python/TS worker's kind could never be enqueued — Enqueue requires a definition and remote handlers live outside the binary. Deployments now declare remote kinds at boot with a comma list of <kind>[:<language>] pairs; compose passes the variable through. Also retires RegisterRemote's stale pre-dispatch doc comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * docs(jobs): retire stale pre-R3 dispatcher narrative in package doc Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * fix(modules): actually construct the workload-module manager at boot server.Deps documented the contract (main constructs via modules.Load + ApplyMigrations before New) but nothing called it — Deps.Modules was always nil, so no /api/v1/workload routes ever mounted. Wire it with the privileged migrate connection for module DDL, mirroring how River's migrations run when the serving role is restricted. Found live: compose stack with AF_STACK_WORKLOAD_MODULES=notes served 404 on /api/v1/workload/notes/notes and /api/v1/admin/modules listed nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * fix(jobs): worker lease honors platform scope semantics The lease guard required the literal jobs:work scope string, rejecting full-access keys (empty scopes / "*" / "admin") and bare "jobs" area grants that the platform scope model (scopes.go) accepts everywhere else. Reuse scopeSatisfied so the two rule sets cannot diverge. Found live: a legacy full-access key got 403 SCOPE_REQUIRED on lease. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * chore(jobs): drop now-unused scopesContain helper Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * feat(cli): wire af-stack module validate + agent validate subcommands R4 listed both, but only 'new' was dispatched and the validate package was reachable only through 'af-stack test'. Add the standalone subcommands (same validate package, so the command and the test gate share one contract), honoring --json and the documented exit codes. Also fix the top-level unknown-command path to return ExitUsage (2) instead of the generic 1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * fix(sdk): agent-call result field — SDKs modeled output, runtime returns result Live SDK conformance against the running stack caught a real SDK<->runtime drift (exactly the R1/R6 gap): a supportdesk.echo call returns {status: succeeded, result: {...}} but both SDKs typed the value as 'output', which the runtime never sends — so the documented CallResult.output was always null/undefined. - sdk-py CallResult: add typed 'result', keep 'output' as a back-compat alias, mirror the two in a model_validator - sdk-ts callResultSchema + executionStatusResponseSchema: add 'result'; add 'result' to OPAQUE_FIELDS so the camelizer never renames the agent's return keys, and mirror result<->output in camelEnvelope - conformance harnesses assert on result (accepting output) — they encoded the same wrong assumption Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * fix(worker-conformance): conf-permanent expects River 'cancelled', not 'discarded' PermanentError maps to river.JobCancel, whose terminal state is 'cancelled'; 'discarded' is River's exhausted-retry dead-letter state. Verified live: both the py and ts reference workers reach 'cancelled' on a permanent failure. Both suites now pass 5/5 against a live stack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> * ci(lint): compute golangci new-issues from git, not the diff API The golangci-lint-action's only-new-issues mode fetches the PR patch from the GitHub diff API to decide which findings are new. That API has a hard 20k-line ceiling; a large PR (this one is ~27k lines) gets 'diff too large', the action's new-issue detection fails, and it falls back to reporting the entire ~106-issue pre-existing backlog — failing the gate on debt the PR never touched. Switch to golangci-lint's native --new-from-merge-base=origin/main, which computes the exact same 'only new issues' set from git history (fetch-depth: 0 already provides origin/main) with no size ceiling. The documented gate intent is unchanged: the backlog stays un-gated, any issue a PR adds on its own changed lines still fails the build. Verified locally that this PR yields 0 new issues under this flag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> --------- Signed-off-by: Abir Abbas <abirabbas1998@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 89b5a10 commit 034e207

183 files changed

Lines changed: 26224 additions & 641 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,16 @@ jobs:
112112
# v2.5.0's release binary is built with go1.25.1; older v2 binaries
113113
# (e.g. v2.1.6, built with go1.24) refuse a config targeting go 1.25.
114114
version: v2.5.0
115-
only-new-issues: true
115+
# Compute NEW issues from git, not the GitHub diff API. The action's
116+
# only-new-issues path fetches the PR patch from the API, which has a
117+
# hard 20k-line ceiling ("diff too large") — a large PR then trips it
118+
# and the action falls back to reporting the entire ~106-issue
119+
# backlog. `--new-from-merge-base` does the identical filtering via
120+
# git (fetch-depth: 0 above provides origin/main), with no ceiling:
121+
# only issues on lines this PR changed relative to the merge base
122+
# fail the build. Same gate, robust to PR size.
123+
only-new-issues: false
124+
args: --new-from-merge-base=origin/main
116125

117126
test-go:
118127
name: Test (Go)

‎Makefile‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
.PHONY: help preflight dev test test-go test-py test-ts lint lint-go lint-py lint-ts \
2+
lint-migrations \
3+
sdk-conformance worker-conformance \
24
build build-go build-cli build-runtime build-dashboard build-images \
35
install-cli smoke-cli \
46
up down logs clean install-deps fmt
@@ -17,6 +19,8 @@ help:
1719
@echo " make test-go Run Go tests"
1820
@echo " make test-py Run Python tests"
1921
@echo " make test-ts Run TypeScript tests"
22+
@echo " make sdk-conformance Live SDK conformance (py+ts) vs a running stack"
23+
@echo " make worker-conformance Live worker-protocol conformance vs a running stack"
2024
@echo " make lint Run all linters"
2125
@echo " make fmt Auto-format all code"
2226
@echo " make build Build all artifacts (CLI + runtime + dashboard)"
@@ -63,14 +67,41 @@ test-ts:
6367
@echo "==> Running TypeScript tests"
6468
@command -v pnpm >/dev/null && pnpm -r test 2>/dev/null || echo "(no TS tests yet)"
6569

70+
# ─── Live conformance harnesses (run against a RUNNING stack) ──────────────
71+
# Config: BASE_URL (default below) and API_KEY (optional in personal mode).
72+
# Both harnesses emit a JSON summary on stdout and exit non-zero on any FAIL.
73+
BASE_URL ?= http://localhost:8080
74+
75+
# sdk-conformance runs the Python AND TypeScript SDK harnesses back-to-back,
76+
# proving both SDKs speak the same live wire contract (see scripts/sdk-conformance).
77+
sdk-conformance:
78+
@echo "==> SDK conformance (Python)"
79+
@BASE_URL="$(BASE_URL)" API_KEY="$(API_KEY)" python3 scripts/sdk-conformance/run.py
80+
@echo "==> SDK conformance (TypeScript)"
81+
@pnpm --filter @af-stack/sdk build >/dev/null
82+
@BASE_URL="$(BASE_URL)" API_KEY="$(API_KEY)" sh -c 'if command -v tsx >/dev/null 2>&1; then tsx scripts/sdk-conformance/run.ts; else node --experimental-strip-types scripts/sdk-conformance/run.ts; fi'
83+
84+
# worker-conformance enqueues the spec.json vector jobs and verifies their
85+
# terminal states against a reference worker (see scripts/worker-conformance).
86+
# WORKER=py|ts selects the reference worker language (default py).
87+
worker-conformance:
88+
@echo "==> Worker conformance (WORKER=$(or $(WORKER),py))"
89+
@BASE_URL="$(BASE_URL)" API_KEY="$(API_KEY)" WORKER="$(or $(WORKER),py)" bash scripts/worker-conformance/run.sh
90+
6691
lint: lint-go lint-py lint-ts
6792
@echo "==> Lint passed"
6893

69-
lint-go:
94+
lint-go: lint-migrations
7095
@echo "==> Linting Go"
7196
@command -v golangci-lint >/dev/null && golangci-lint run ./... 2>/dev/null || echo "(golangci-lint not installed or no Go yet)"
7297
@go vet ./... 2>/dev/null || echo "(no Go packages yet)"
7398

99+
# Static safety lint for goose migrations (Up/Down present, no unmarked
100+
# destructive Up ops, balanced StatementBegin/End). Fails hard on any finding.
101+
lint-migrations:
102+
@echo "==> Linting migrations"
103+
@go run ./services/runtime/cmd/migrationlint
104+
74105
lint-py:
75106
@echo "==> Linting Python"
76107
@command -v ruff >/dev/null && ruff check packages/sdk-py 2>/dev/null || echo "(ruff not installed or no Python yet)"

‎deploy/prometheus/README.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# BackAI Prometheus alert rules
2+
3+
`alerts.yml` ships the default alert rules for the R7 production operating
4+
contract. They cover readiness flapping, LLM provider failure rate, background
5+
job queue backlog, budget-enforcement rejection spikes, outbound webhook
6+
delivery failures, Postgres pool saturation, and backup-test staleness.
7+
8+
Every `backai_*` metric referenced is exported by the runtime at `/metrics`
9+
(registered in `services/runtime/internal/appmetrics`). A Go test
10+
(`TestAlertsReferenceOnlyExistingMetrics`) fails the build if a rule ever
11+
references a metric the runtime doesn't export, so these rules can't rot.
12+
13+
## Wiring
14+
15+
Plain Prometheus:
16+
17+
```yaml
18+
# prometheus.yml
19+
rule_files:
20+
- /etc/prometheus/alerts.yml # mount deploy/prometheus/alerts.yml here
21+
22+
scrape_configs:
23+
- job_name: backai-runtime # the readiness rules assume this job name
24+
metrics_path: /metrics
25+
static_configs:
26+
- targets: ["runtime:9090"] # the runtime metrics listener (AF_STACK_METRICS_ADDR)
27+
```
28+
29+
The `up{job="backai-runtime"}` selector in the readiness rules assumes the
30+
scrape job is named `backai-runtime`. If you scrape under a different job name
31+
(e.g. via the Prometheus Operator `ServiceMonitor` in
32+
`deploy/helm/af-stack`), update that selector to match.
33+
34+
## Validating locally
35+
36+
```bash
37+
promtool check rules deploy/prometheus/alerts.yml # if promtool is installed
38+
go test ./services/runtime/internal/appmetrics/... # structural + metric-existence check
39+
```
40+
41+
## Notes
42+
43+
- The `/metrics` endpoint is served on its own listener
44+
(`AF_STACK_METRICS_ADDR`, default `:9090`) so per-tenant cost/usage labels are
45+
never exposed on the public API port. Restrict it to your Prometheus.
46+
- `BackAIBackupTestStale` only fires once at least one successful backup test
47+
has been recorded (`backai_backup_test_last_success_timestamp > 0`), so it
48+
stays quiet when `BACKUP_TEST_ENABLED` is off.

‎deploy/prometheus/alerts.yml‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# BackAI default Prometheus alert rules (R7 production operating contract).
2+
#
3+
# Load these into Prometheus via `rule_files:` (see deploy/prometheus/README.md).
4+
# Every `backai_*` metric referenced here is registered by the runtime
5+
# (services/runtime/internal/appmetrics) and asserted to exist by
6+
# TestAlertsReferenceOnlyExistingMetrics — an alert can never reference a metric
7+
# the runtime doesn't export.
8+
#
9+
# The `up` series is synthesised by Prometheus per scrape target. The selector
10+
# `job="backai-runtime"` is the assumed scrape-job name for the runtime's
11+
# /metrics endpoint — adjust it to match your scrape config / ServiceMonitor.
12+
groups:
13+
- name: backai-readiness
14+
rules:
15+
- alert: BackAIRuntimeDown
16+
expr: up{job="backai-runtime"} == 0
17+
for: 2m
18+
labels:
19+
severity: critical
20+
annotations:
21+
summary: "BackAI runtime target is down"
22+
description: "The runtime /metrics target {{ $labels.instance }} has been unscrapeable for 2m."
23+
- alert: BackAIReadinessFlapping
24+
expr: changes(up{job="backai-runtime"}[10m]) > 4
25+
for: 0m
26+
labels:
27+
severity: warning
28+
annotations:
29+
summary: "BackAI runtime readiness is flapping"
30+
description: "Target {{ $labels.instance }} changed up/down state more than 4 times in 10m — readiness is unstable (check /ready envelope for a prodcheck_code or db_unavailable)."
31+
32+
- name: backai-llm
33+
rules:
34+
- alert: BackAILLMProviderFailureRateHigh
35+
expr: |
36+
sum(rate(backai_llm_requests_total{status="error"}[5m]))
37+
/ clamp_min(sum(rate(backai_llm_requests_total[5m])), 1) > 0.2
38+
for: 10m
39+
labels:
40+
severity: warning
41+
annotations:
42+
summary: "BackAI LLM gateway error rate above 20%"
43+
description: "More than 20% of LLM gateway requests have failed over the last 5m — a provider or the LiteLLM proxy is likely degraded."
44+
45+
- name: backai-jobs
46+
rules:
47+
- alert: BackAIJobsQueueBacklog
48+
expr: backai_jobs_queue_oldest_age_seconds > 900
49+
for: 10m
50+
labels:
51+
severity: warning
52+
annotations:
53+
summary: "BackAI background job queue is backed up"
54+
description: "The oldest past-due background job has waited over 15m for a worker — the jobs manager may be stalled or under-provisioned."
55+
56+
- name: backai-budget
57+
rules:
58+
- alert: BackAIBudgetRejectionsSpike
59+
expr: sum(rate(backai_budget_rejections_total[5m])) > 1
60+
for: 5m
61+
labels:
62+
severity: warning
63+
annotations:
64+
summary: "BackAI budget-enforcement rejections spiking"
65+
description: "Budget enforcement is rejecting more than 1 LLM call/s (402) over 5m — a tenant or key has hit its cap, or a budget is misconfigured."
66+
67+
- name: backai-webhooks
68+
rules:
69+
- alert: BackAIWebhookDeliveryFailureRateHigh
70+
expr: |
71+
sum(rate(backai_webhook_deliveries_total{result="failed"}[10m]))
72+
/ clamp_min(sum(rate(backai_webhook_deliveries_total[10m])), 1) > 0.5
73+
for: 15m
74+
labels:
75+
severity: warning
76+
annotations:
77+
summary: "BackAI outbound webhook delivery failure rate above 50%"
78+
description: "More than half of outbound webhook deliveries have failed over 10m — subscriber endpoints may be down or misconfigured."
79+
80+
- name: backai-db
81+
rules:
82+
- alert: BackAIDBConnectionSaturation
83+
expr: |
84+
backai_db_pool_acquired_connections
85+
/ clamp_min(backai_db_pool_max_connections, 1) > 0.9
86+
for: 5m
87+
labels:
88+
severity: warning
89+
annotations:
90+
summary: "BackAI Postgres pool near saturation"
91+
description: "The runtime Postgres pool has been over 90% of max connections for 5m — queries may start queueing; raise AF_STACK_DATABASE_MAX_CONNECTIONS or investigate slow queries."
92+
93+
- name: backai-backup
94+
rules:
95+
- alert: BackAIBackupTestStale
96+
expr: |
97+
backai_backup_test_last_success_timestamp > 0
98+
and (time() - backai_backup_test_last_success_timestamp) > 129600
99+
for: 0m
100+
labels:
101+
severity: critical
102+
annotations:
103+
summary: "BackAI backup/restore verification is stale"
104+
description: "No successful backup/restore test has completed in over 36h — restore integrity is unverified. Check the backup-test cron (BACKUP_TEST_ENABLED)."

‎docker-compose.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,14 @@ services:
120120
AF_STACK_DATABASE_URL: postgres://afstack_app:${AF_STACK_APP_DB_PASSWORD:-afstack_app}@postgres:5432/afstack?sslmode=disable
121121
AF_STACK_MIGRATE_DATABASE_URL: postgres://afstack:afstack@postgres:5432/afstack?sslmode=disable
122122
AF_STACK_AGENTFIELD_URL: http://agentfield:8080
123+
# Workload modules (PRD R2). The path matches the ro mount below;
124+
# the enabled list is empty by default — the shipped notes module
125+
# is a reference, opt in with AF_STACK_WORKLOAD_MODULES=notes.
126+
WORKLOAD_MODULES_PATH: /app/workload-modules
127+
AF_STACK_WORKLOAD_MODULES: ${AF_STACK_WORKLOAD_MODULES:-}
128+
# Remote (Python/TS) job kinds served by external pull-workers (R3):
129+
# comma list of <kind>[:<language>], e.g. "index-document:python".
130+
AF_STACK_REMOTE_JOB_KINDS: ${AF_STACK_REMOTE_JOB_KINDS:-}
123131
# KMS for the secrets vault (envelope encryption). Override in
124132
# production with `openssl rand -hex 32`.
125133
AF_STACK_AUTH_SECRET: ${AF_STACK_AUTH_SECRET:-dev-secret-change-me-in-prod}
@@ -216,6 +224,12 @@ services:
216224
# at the runtime's expected lookup path. Optional — when missing
217225
# the runtime falls back to defaults + env overrides.
218226
- ./apps/backend/config.yaml:/config.yaml:ro
227+
# Declarative workload modules (PRD R2): the runtime discovers
228+
# backai.module.yaml manifests here, applies their migrations and
229+
# serves tenant-scoped CRUD under /api/v1/workload/<id>/. Enable a
230+
# discovered module with AF_STACK_WORKLOAD_MODULES=<id>[,<id>...]
231+
# (or enabled: true in its manifest).
232+
- ./workload-modules:/app/workload-modules:ro
219233
# Docker socket for the sandbox adapter. Without this mount the
220234
# sandbox falls back to "unconfigured" and run endpoints return 503.
221235
# In production, prefer gVisor or e2b for stronger isolation.

0 commit comments

Comments
 (0)