This repository was archived by the owner on Jun 7, 2026. It is now read-only.
feat(gateway): add concurrency limits for request handling - #65
Merged
Conversation
Implement global concurrency limits to prevent resource exhaustion: - Separate limits for streaming (default: 3) and non-streaming (default: 5) - Configurable via KOINE_MAX_STREAMING_CONCURRENT and KOINE_MAX_NONSTREAMING_CONCURRENT - Returns 429 Too Many Requests when limit exceeded - Health endpoint now includes concurrency status Files: - Add concurrency.ts with acquire/release semaphore and wrapper - Wrap generate and stream routes with concurrency limiter - Update types.ts with CONCURRENCY_LIMIT_ERROR code - Add concurrency tests
- Add KOINE_MAX_STREAMING_CONCURRENT and KOINE_MAX_NONSTREAMING_CONCURRENT to environment variables documentation - Document 429 response in API reference - Add retry handling examples to both SDK READMEs and sdk-guide - Update CONTRIBUTING.md roadmap (mark concurrency limits done, remove request queuing and per-key throttling as out of scope) - Add concurrency config section to docker-deployment guide - Mention concurrency limits in root README features
Address issues found by code review: - Add env var validation with logged fallback for invalid values (prevents NaN from silently disabling all limits) - Add logging when 429 concurrency limit is hit - Add validation to setConfig (throws on invalid values) - Add logging in catch block before re-throw - Update comments for clarity (JS event loop, not Bun-specific) - Add tests for error handling, validation, and streaming integration
Address remaining code review findings: - Add Zod refinements to concurrencyPoolSchema (int, nonnegative, active <= limit) - Document that getConfig() returns a shallow copy to prevent mutation - Add test for slot exhaustion behavior (rejects when slots full) - Add test for response close/finish event handling
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Retry-After: 5header when limits exceededKOINE_MAX_STREAMING_CONCURRENT,KOINE_MAX_NONSTREAMING_CONCURRENTChanges
Core Implementation
packages/gateway/src/concurrency.ts- Semaphore-based limiter with acquire/release patternpackages/gateway/src/types.ts- AddedCONCURRENCY_LIMIT_ERRORcode and health schemawithConcurrencyLimit()middlewareValidation & Logging
setConfig()validates inputs are non-negative integersDocumentation
docs/environment-variables.md,docs/api-reference.md,docs/sdk-guide.mdCONTRIBUTING.mdroadmapTests (24 new tests)
Test plan
bun run test- 107 tests passbun run lint- No errorsdocker compose upand curl