perf: parallelize the pipeline + add a Groq rate limiter - #7
Merged
Conversation
…iter three stages ran fully sequentially and dominated wall-clock time. the runner was already concurrent; these were not: - ontology.build_anchors: ~16 calls one at a time -> gather, bounded by a semaphore. pulled the parse logic into a _parse_anchors staticmethod. - generator.generate_test_cases: 4 buckets one at a time -> gather the 4. - judge: cli looped `await judge_single` per test -> new judge_all() gathers with a semaphore, cli builds DiffResults from the ordered results. behavior is identical, just concurrent. also added a token-bucket rate limiter (diffprompt/ratelimit.py) on every groq request so the new concurrency doesnt burst past the free-tier RPM cap. semaphores bound concurrency; this bounds rate, which is the thing that actually trips 429s. tune with DIFFPROMPT_GROQ_RPM (default 30, 0 disables). tests: test_ratelimit, test_judge, test_ontology + a generator bucket test. 38 passed locally. closes #6 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6.
makes the pipeline actually fast. the runner was already concurrent; three other stages ran fully sequentially and ate all the wall-clock time.
whats parallel now
ontology.build_anchors- was ~16 LLM calls one at a time. now gathered, bounded by a semaphore. parse logic pulled into a_parse_anchorsstaticmethod.generator.generate_test_cases- 4 buckets were sequential. now the 4 run concurrently via gather.cli.pyloopedawait judge_singleonce per test (the biggest cost). newjudge_all()gathers them with a semaphore; cli buildsDiffResults from the ordered results.behavior is identical - same results, same order, just concurrent.
rate limiter
the new concurrency would happily burst past Groq's free-tier RPM and eat 429s. semaphores bound concurrency, not rate, so they dont help here. added a token-bucket limiter (
diffprompt/ratelimit.py) on every Groq request - that bounds rate, which is the thing that actually trips the limit.DIFFPROMPT_GROQ_RPM=0disables it, higher values for paid tierstests
test_ratelimit.py- disabled no-op, initial burst, pacing after draintest_judge.py- judge_all preserves order + counttest_ontology.py- build_anchors covers every tag (fake embedder, no model load) + _parse_anchors38 passed locally (was 30).
note
the queue-vs-gather question came up. for a fixed fan-out (known set of calls, collect results) gather+semaphore is simpler and equivalent. a queue/worker-pool earns its keep once we add streaming/continuous monitoring - not here.