Conversation
Ricky-7-Yan
left a comment
There was a problem hiding this comment.
Reviewed the follow-up commit independently. I also checked the model registry shipped with this repository's pinned LiteLLM 1.98.0 (rather than relying on the newer 1.100.0 result): it reports supports_xhigh_reasoning_effort=false for gpt-5, gpt-5-mini, gpt-5.1, and gpt-5.1-codex, and true for gpt-5.2, gpt-5.5, and gpt-5.1-codex-max. The metadata-driven clamp therefore matches the runtime dependency, while the missing-key/lookup-error fallback avoids downgrading future unknown models. The parameterized test covers both decisions plus prefix and _thinking normalization. Approving the logic. The branch currently conflicts after #3259's squash merge, so it still needs a rebase/drop of the two stacked commits before it can merge.
78eb004 to
054489e
Compare
|
Rebased onto @Ricky-7-Yan was right that the branch had gone stale: #3259 landed the Nothing in the clamp changed in the rebase; the conflict was entirely in the lines #3259 had already taken.
|
IsmaelMartinez
left a comment
There was a problem hiding this comment.
Thanks for turning the rebase round so fast after Ricky's note, and for reading the registry rather than listing model names.
One thing to redo, and it is not yours: #2999 landed on main about ninety minutes after you rebased and moved this block onto helper methods, so model_base no longer exists in that scope. It text-merges cleanly and the green checks predate it, so nothing on this page shows it. Merged onto today's main the suite is 4 failed with name 'model_base' is not defined.
Suggestion inline. With it the suite is green, and reverting the handler still takes all four new cases red.
| "using 'xhigh' for reasoning_effort='max'" | ||
| ) | ||
| effort = ReasoningEffort.XHIGH.value | ||
| lookup_model = model_base.removesuffix('_thinking') |
There was a problem hiding this comment.
After #2999 the model reaching this block carries the routed provider prefix, so the lookup name has to be stripped here. Applies and runs green on this head too.
| lookup_model = model_base.removesuffix('_thinking') | |
| lookup_model = model | |
| while lookup_model.startswith(("openai/", "azure/")): | |
| lookup_model = lookup_model.removeprefix("openai/").removeprefix("azure/") | |
| lookup_model = lookup_model.removesuffix("_thinking") |
Closes #3271. Stacks on #3259 — the first two commits here are that PR's; the third is this change. Reviewing the last commit alone (
78eb004) gives the whole of this fix, and this becomes a one-commit diff once #3259 lands.#3259 translates this project's
maxalias toxhighon the direct GPT-5 path. As #3271 notes, that is right for GPT-5.2 and later and wrong for the two older models, which do not acceptxhigheither: it swaps one rejected value for another. This clamps tohighwhere litellm saysxhighis unsupported.Registry reading, verified locally on litellm 1.100.0, identical under the fetched and the bundled map:
supports_xhigh_reasoning_effortSo the set is wider than the issue names —
gpt-5-miniandgpt-5.1-codexare affected too, which is the argument for reading the metadata rather than listing model names.What it does. On
reasoning_effort='max'down the GPT-5 branch, it looks the model up withlitellm.get_model_info(stripping a_thinkingsuffix, which the surrounding code already treats as a decoration rather than part of the model id) and sendshighonly when the registry answersFalseoutright.True, a missing key, and a lookup that raises all keepxhigh: an unknown model is not evidence thatxhighis unsupported, and silently downgrading a model litellm has not heard of yet would be the same class of bug in the other direction.Test plan.
test_gpt5_reasoning_effort_max_uses_xhigh_metadatais parametrised over the four cases — supported, unsupported, key absent, lookup raises — and asserts both the effort that goes out and the exact string the lookup was made with, so the_thinkingstripping and the provider-prefix stripping are pinned rather than incidental.PYTHONPATH=. uv run pytest tests/unittest/test_litellm_reasoning_effort.py -q— 245 passeduv run ruff checkon both touched files — clean