Repository navigation
Add configurable per-request LLM token and prompt-byte limits - #1026
SajjadPSavoji wants to merge 4 commits into
Conversation
Signed-off-by: Sajjad Pakdamansavoji <44399382+SajjadPSavoji@users.noreply.github.com>
e53a509 to
758e0c0
Compare
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Decision: Changes Requested
Value and readiness: Per-deployment request ceilings are useful, and the documentation distinguishes prompt text from serialized HTTP size. This head still submits requests above valid configured limits, so it does not yet deliver its intended outcome.
Material findings — required:
src/skillspector/llm_analyzer_base.py:1425,1450raises the content allowance to at least 1,024 even when SKILLSPECTOR_MAX_BATCH_INPUT_TOKENS is, for example, 100. A short-line source around 600 estimated tokens can therefore be retained as one batch and submitted above 100; the new sync/async preflight at:1552-1571checks bytes only. Fixed wrapper/schema overhead can also exhaust a tiny token allowance. Respect an exhausted allowance and check the complete built input before both invocation modes, preserving an explicit SIZE_LIMIT failure without a provider call. Add tiny-token and fixed-overhead regressions through normal run paths; the new 5,000-token property assertion does not exercise this boundary.- The final byte guard at
:1552-1557checks raw prompt bytes._fixed_prompt_bytesincludes CLI schema instructions, but a nonpositive resulting byte allowance at:1454-1459disables chunk byte limiting and only warns at:1474+. With a tiny file and a cap between the raw prompt and raw-plus-schema sizes, the guard passes;_StructuredAgentCLIModel._completeinllm_utils.py:345-353appends the schema and submits over the configured cap. Refuse irreducible augmented overhead or validate the actual augmented text before submission. Add a regression where raw text fits and schema augmentation alone exceeds the limit; the existing test instead makes the raw base prompt too large. _tp4_batch_token_ceilinginmcp_tool_poisoning.py:90-100can return a configured value above 32,000, and:1267then raises the old TP4 ceiling when the model/global budgets are larger. This contradicts the README and description's promise that each setting only lowers a limit. Clamp to the existing TP4 ceiling, or explicitly revise the intended contract and test the accepted larger-value policy.
PIC tradeoffs: The byte control deliberately concerns prompt text, not HTTP serialization. That is reasonable, but text added by the CLI transport is still prompt text and must obey it. The requested fixes do not require changing model defaults or output budgets.
Verification and gaps: Reviewed the full five-file diff, pinned planner, structured CLI transport, sync/async/recovery paths, TP4 planner and test assertions at 758e0c062c1f32bc62e8a46b5c3b85a13cd6f664; an independent static reviewer confirmed these paths. All six hosted checks succeeded, but the boundary cases above are absent. Findings are static control-flow proofs, not executed reproductions. No contributor code/tests/imports/builds/setup, regexes or provider calls were executed locally.
|
updated. The configured token and byte caps now go through the existing |
Signed-off-by: Sajjad Pakdamansavoji <44399382+SajjadPSavoji@users.noreply.github.com>
…ed input Signed-off-by: Sajjad Pakdamansavoji <44399382+SajjadPSavoji@users.noreply.github.com>
5e7f3a6 to
112c532
Compare
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Decision: Changes Requested
Value and readiness
Configurable input limits help gateways with tighter request constraints. The earlier normal-path token cap, schema-only rejection and TP4 ceiling corrections are present. The new TP4 planner can still skip source that fits the configured cap.
Material findings
Required — src/skillspector/nodes/analyzers/mcp_tool_poisoning.py:1343–1354. The planner subtracts the fixed header and then another 4,096 bytes from every byte allowance. For a small one-line candidate, set the byte cap to the actual size of its built prompt plus the TP4 schema instructions. This fits the shared preflight byte check exactly, but the extra 4,096-byte reserve exceeds the small schema appendix and line combined. The planner clamps the code allowance to zero and skips every line before the normal request guard can assess it. This yields a SIZE_LIMIT partial scan for an input that fits the documented cap. Compute the allowance from the same augmented prompt the invocation path checks, or refine the chunk with the exact predicate before declaring it too large. Add a full TP4 regression with the unmodified margin, a tight cap that fits and one that genuinely cannot fit.
The generic planner falls back to its full prompt predicate when the same coarse size estimate is exhausted. The added TP4 tests cover a larger cap and an artificial negative margin; neither covers this positive small-cap case.
PIC tradeoffs
None identified. The user-configured safety limit can be enforced without reserving unused capacity that prevents valid inspection.
Verification and gaps
Read the five-file diff and exact-head generic planner, TP4 prompt construction and shared preflight. Inspected the sync/async rejection, line-fragment and TP4 coverage tests and the author’s response. Six hosted checks passed. No contributor code or tests were executed locally.
Reviewed head: 112c5327d2b3a7616a30f2d84add062fe7db9061.
Signed-off-by: Sajjad Pakdamansavoji <44399382+SajjadPSavoji@users.noreply.github.com>
|
fixed, the TP4 byte allowance now comes from the same language and schema augmented text the preflight checks, with no extra 4 KB reserve. The negative-margin test is gone. In its place there's a full TP4 run where a cap equal to the submitted input sends everything and one byte less marks only that file |
Deployments with a gateway that rejects big requests had to patch the code to get a smaller per-request size. This adds three optional env vars:
SKILLSPECTOR_MAX_BATCH_INPUT_TOKENS,SKILLSPECTOR_MAX_PROMPT_BYTESandSKILLSPECTOR_TP4_MAX_BATCH_INPUT_TOKENS. Each one only lowers a limit, so the model budget and the CLI provider byte cap still win when they're smaller. Unset or blank keeps today's behavior and bad values log a warning without echoing the value.The byte cap goes through the existing chunking in
get_batchesand TP4's own chunk planner. Prompts are also checked right before sending in the sync and async paths. A line or fixed prompt that can't fit is markedSIZE_LIMITfor that batch, nothing is truncated and other batches keep running. README has a short section on units and precedence.Thanks @SajjadPSavoji for the suggested design in the thread.
Tests: new
TestConfigurableRequestLimitsintests/nodes/test_llm_analyzer_base.pyplus the TP4 token ceiling, byte ceiling andSIZE_LIMITpartial tests intests/test_mcp_tool_poisoning.py. They fail before the change and pass after. I also ran the rest oftest_llm_analyzer_base.py,test_mcp_tool_poisoning.py,test_meta_analyzer.pyandtest_meta_analyzer_fallback.py, andruff checkon the touched files.Fixes #1003