Skip to content

feat: daily budget throttling with rate-aware proxy sleep#469

Merged
BYK merged 1 commit into
mainfrom
feat-budget-throttle
May 24, 2026
Merged

feat: daily budget throttling with rate-aware proxy sleep#469
BYK merged 1 commit into
mainfrom
feat-budget-throttle

Conversation

@BYK
Copy link
Copy Markdown
Owner

@BYK BYK commented May 24, 2026

Summary

Add a configurable daily USD spending cap to the Lore gateway. When the budget is approached, the gateway applies an invisible proxy-level sleep before forwarding requests upstream — progressively slowing agents rather than hard-blocking them.

How It Works

  1. Set budget via the UI dashboard (/ui/costs) or env var LORE_DAILY_BUDGET. Persisted in SQLite kv_meta table — survives restarts, easy to tune.

  2. Rate detection: Cost-rate EMA (α=0.15) tracks spending velocity across all sessions. Time-gap-adjusted alpha means long idle gaps naturally decay the EMA.

  3. Throttle curve: 60s × pressure² × tanh(overshoot/3) — smooth, C∞ continuous. Starts at sub-second delays at 60% budget, maxes at 60s under extreme overshoot. No cliff edges.

  4. Invisible proxy sleep: The agent doesn't know it's being throttled — the upstream just appears to take a little longer. A 3-5s sleep is imperceptible during a normal coding turn.

  5. Cache TTL safety cap: Sleep is capped at 50% of remaining cache TTL window to prevent cache busts that would increase costs.

Throttle Behavior

Spend % Rate vs Target Delay
<50% any 0s (no throttle)
60% ~0.8s
70% ~3.1s
80% ~6.9s
80% ~19.2s
95% ~36.9s

Configuration

  • UI: Set/disable budget from the Costs page (/ui/costs)
  • Env var: LORE_DAILY_BUDGET=10.00 (overrides DB value)
  • Resolution: env var > DB > 0 (disabled)
  • Default: 0 (disabled, zero overhead on hot path)

Dashboard

  • Global: Budget progress bar with spend/budget ratio, current rate, throttle event count
  • Per-session: Throttle event count + total delay when throttled

What's NOT Throttled

  • Meta requests (title generation, etc.) — cheap
  • Compaction requests — intercepted locally
  • Worker calls — their cost IS counted, but they're not delayed (have their own rate limiting)

Files Changed

File Changes
packages/gateway/src/cost-tracker.ts Daily spend accumulator, cost-rate EMA, throttle curve, DB-backed budget, pre-request cost estimator
packages/gateway/src/pipeline.ts Throttle interception before forwardToUpstream() with cache TTL safety cap
packages/gateway/src/server.ts Bootstrap daily spend from DB on startup
packages/gateway/src/ui.ts Budget settings form, progress bar, throttle diagnostics
packages/gateway/test/budget-throttle.test.ts 28 tests covering all throttle zones, monotonicity, smoothness, EMA, accumulation

No new files (except test). No DB schema changes. No core package changes.

@BYK BYK force-pushed the feat-budget-throttle branch from f2e724a to 643fd12 Compare May 24, 2026 17:35
@BYK BYK merged commit 9afaaae into main May 24, 2026
7 checks passed
@BYK BYK deleted the feat-budget-throttle branch May 24, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant