perf: size the inference thread pool from the cgroup, not the host - #10
Merged
Merged
Conversation
…the host Nothing passed --threads, so onnxruntime sized its pool itself. Left alone it counts the cores it can see, and inside a container that is usually the host's core count rather than the cgroup's share of it. Oversubscribing is not a mild loss. Measured on a 4-core machine with the real YOLOX-Tiny at 416x416: 1 thread 112 ms/frame 2 threads 58 ms/frame 4 threads 65 ms/frame 8 threads 194 ms/frame <- 3.5x slower than the best So a service given two vCPUs on a large host can run several times slower than the hardware allows, with nothing to indicate anything is wrong. The worker now defaults to os.availableParallelism(), which is cgroup-aware where os.cpus().length is not, capped at four because more than four only ever cost time here. An explicit --threads still wins. Whether this is a real win on Railway or a no-op is not something that can be measured from outside the container, so the worker now reports both numbers before it starts: threads: using 4 (cgroup-aware 4, visible cores 4) If those two disagree in production, this change mattered. Printed before the run rather than after it: a detection pass takes minutes and can fail, and a diagnostic that only appears on success is no use in either case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ralyodio
added a commit
that referenced
this pull request
Aug 9, 2026
…oducing it (#11) "It says it's done but there's no suggestions." That is unanswerable at the moment, because the two numbers that distinguish the possible causes are computed and then thrown away. analyzeProject returns tracksCreated, momentsGenerated and warnings. The web action calls it as `void analyzeProject(...)`, so all three are discarded. A run that found nothing reported "completed" and nothing else — and zero moments from zero tracks needs an entirely different fix from zero moments from tracks that scored too low. The run now logs what it produced, and when that is nothing, which of the two it was: done: 14 track(s), 0 suggested moment(s) Tracks were found but none scored above the 0.35 threshold for soccer. Marking an athlete to follow gives scoring an anchor and usually raises scores. or done: 0 track(s), 0 suggested moment(s) No tracks were produced, so there was nothing to score. The same applies to the worker's own stderr. Besides progress it reports which thread pool it chose, that it overrode a requested input size, and a closing summary of frames, detections and tracks. run() collects all of it into a string that is only read when the run *fails*, so on a successful run none of it reached anyone — including the threads diagnostic added in #10 specifically to be read. Those lines now become job log entries, where the person watching already is. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ralyodio
added a commit
that referenced
this pull request
Aug 9, 2026
… nothing (#14) Three findings from a real basketball run that produced 1287 tracks and zero suggested moments. Without an athlete to follow, a moment is arithmetically impossible. Three of the seven scoring signals need a focal track and together carry 0.6 of the 1.15 total weight. With no focal track the only signal that can fire is high_motion at 0.1, so the best any window can score is 0.087 against a threshold of 0.35. The app said "Nothing suggested yet — run analysis", ran analysis for a minute, and completed with nothing — never mentioning that it could not have succeeded. It now says so before the work starts, not after. Scoring looked for a track named `goal` whatever the sport. Basketball tracks a `hoop`, hockey a `net`, football an `end_zone`, so for those sports both target signals were permanently dark even with a model that saw the thing perfectly. Soccer worked by luck of naming. Sports now declare the class that is their scoring target, derived from the noun where the two agree and stated explicitly where they do not. A sport with no trackable target resolves to null rather than to a name nothing will match. The thread cap from #10 was tuned on a four-core laptop and production turned out to have twenty-four: worker: threads: using 4 (cgroup-aware 24, visible cores 48) Those two numbers disagreeing is what #10 existed to reveal, and it was right to pin the pool — onnxruntime would have sized from 48. But four is now the limit rather than the protection, so the cap moves to eight, which is where a model this small stops benefiting rather than the size of the machine it was measured on. REELEEL_CV_THREADS overrides it without a deploy, so the ceiling can be found without another release. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Nothing passed
--threads, so onnxruntime sized its own pool. Left alone it counts the cores it can see — and inside a container that is usually the host's core count, not the cgroup's share of it.Oversubscribing is not a mild loss. Measured on a 4-core machine with the real YOLOX-Tiny at 416×416:
A service given two vCPUs on a large host can therefore run several times slower than the hardware allows, with nothing anywhere to indicate a problem.
The change
Default to
os.availableParallelism()— cgroup-aware, whereos.cpus().lengthis not — capped at four, because beyond four every extra thread only cost time here. An explicit--threadsstill wins.Honest about what this is
I cannot tell from outside the container whether this is a real win on Railway or a no-op. It depends entirely on whether the container sees host cores. So the worker now reports both numbers before it starts:
If those two disagree in production, this change mattered and we will be able to see by how much. If they agree, it changed nothing and costs nothing.
Printed before the run rather than after: a detection pass takes minutes and can fail, and a diagnostic that only appears on success is no use in either case. (I wrote it the wrong way round first.)
Verification
380 tests pass — the full suite including the detector integration tests against real weights — eslint and typecheck clean.
The bigger lever, for the record
Threads are worth fixing but
frameStridedominates. For a 45-minute half at the measured rate:balanced)fast)ByteTracker interpolates between sampled frames, so for finding highlights rather than frame-accurate analytics,
fastis a 2.5× win available with no code change at all.