Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions docs/docs/usage-guide/changing_a_model.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,15 +184,21 @@ To use xAI's models with PR-Agent, set:

```toml
[config] # in configuration.toml
model = "xai/grok-2-latest"
fallback_models = ["xai/grok-2-latest"] # or any other model as fallback
model = "xai/grok-4.6"
fallback_models = ["xai/grok-4.6"] # or any other model as fallback

[xai] # in .secrets.toml
key = "..." # your xAI API key
```

You can obtain an xAI API key from [xAI's console](https://console.x.ai/) by creating an account and navigating to the developer settings page.

Grok 4.5 and Grok 4.6 are registered with a 500K token context window (`xai/grok-4.5`, `xai/grok-4.5-latest`, `xai/grok-build-latest`, `xai/grok-4.6`, `openrouter/x-ai/grok-4.5`, `openrouter/x-ai/grok-4.6`). xAI publishes `grok-4.5-latest` and `grok-build-latest` aliases for Grok 4.5; Grok 4.6 currently has no published alias.

Grok 4.5 and Grok 4.6 are always-on reasoning models and honor `config.reasoning_effort` (`low`, `medium`, `high`; `"xhigh"` is supported on Grok 4.6 and later). PR-Agent sends `medium` by default; set `"high"` to restore xAI's native default. This setting is global, so changing it also affects other registered reasoning models. Unsupported values are clamped to the closest accepted level (`none`/`minimal` → `"low"`; `"max"`/`"xhigh"` on Grok 4.5 → `"high"`; `"max"` on Grok 4.6 → `"xhigh"`).

OpenRouter routes (`openrouter/x-ai/grok-4.5`, `openrouter/x-ai/grok-4.6`) apply the same clamping after the OpenRouter effort value is resolved but before the none-versus-budget decision. An explicit `openrouter.reasoning_effort` overrides the global effort; a positive `openrouter.reasoning_max_tokens` remains budget-only and suppresses effort, including a clamped `"none"` value. Routing suffixes such as `:nitro` require `custom_model_max_tokens` because token lookup currently uses exact model IDs.

### Vertex AI

To use Google's Vertex AI platform and its associated models (chat-bison/codechat-bison) set:
Expand Down Expand Up @@ -528,11 +534,11 @@ For `openrouter/...` models you can optionally restrict which upstream providers
# provider_order = ["z-ai", "novita"] # preferred order instead of an allowlist; ignored when provider_only is set
# allow_fallbacks = true # when provider_order is set, allow routing beyond the list
# reasoning_effort = "low" # override global effort: "none", "minimal", "low", "medium", "high", "xhigh" or "max"
# reasoning_max_tokens = 2048 # explicit budget; takes precedence over effort unless effort is "none"
# reasoning_max_tokens = 2048 # explicit budget; ignored only when final effort remains "none"
# max_tokens = 16000 # hard cap on completion tokens for the request
```

`provider_only` and `reasoning_effort = "none"` are useful to pin a specific provider and to bound the cost of reasoning models. Because Openrouter treats effort and token budgets as mutually exclusive, an explicit Openrouter-specific `"none"` keeps reasoning disabled; otherwise a positive `reasoning_max_tokens` value takes precedence over the global effort and other Openrouter-specific values. Invalid Openrouter-specific effort values are warned about and treated as unset, so registered reasoning models fall back to `config.reasoning_effort`. Openrouter normalizes `"max"` to `"xhigh"` in this path to match LiteLLM 1.98.0. Supported effort values vary by model, and models whose metadata marks reasoning as mandatory reject `"none"`. For Anthropic models using a reasoning budget, set the effective output `max_tokens` higher than `reasoning_max_tokens` so the final answer has output headroom. See the Openrouter [provider routing](https://openrouter.ai/docs/guides/routing/provider-selection) and [reasoning tokens](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens) docs.
`provider_only` and `reasoning_effort = "none"` are useful to pin a specific provider and to bound the cost of reasoning models. Because Openrouter treats effort and token budgets as mutually exclusive, an explicit Openrouter-specific `"none"` keeps reasoning disabled when the model supports disabling it. Grok 4.5/4.6 clamp `"none"` before precedence is applied, so a positive budget wins there; otherwise a positive `reasoning_max_tokens` value takes precedence over the global effort and other Openrouter-specific values. Invalid Openrouter-specific effort values are warned about and treated as unset, so registered reasoning models fall back to `config.reasoning_effort`. Openrouter normalizes `"max"` to `"xhigh"` in this path to match LiteLLM 1.98.0. Supported effort values vary by model, and models whose metadata marks reasoning as mandatory reject `"none"`. For Anthropic models using a reasoning budget, set the effective output `max_tokens` higher than `reasoning_max_tokens` so the final answer has output headroom. See the Openrouter [provider routing](https://openrouter.ai/docs/guides/routing/provider-selection) and [reasoning tokens](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens) docs.

### OrcaRouter

Expand Down
20 changes: 20 additions & 0 deletions pr_agent/algo/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,12 @@
'xai/grok-3-mini-beta': 131072,
'xai/grok-3-mini-fast': 131072,
'xai/grok-3-mini-fast-beta': 131072,
"xai/grok-4.5": 500000, # 500K context, but may be limited by config.max_model_tokens
"xai/grok-4.5-latest": 500000,
"xai/grok-build-latest": 500000,
"xai/grok-4.6": 500000, # 500K context, but may be limited by config.max_model_tokens
"openrouter/x-ai/grok-4.5": 500000,
"openrouter/x-ai/grok-4.6": 500000,
'ollama/llama3': 4096,
'watsonx/meta-llama/llama-3-8b-instruct': 4096,
"watsonx/meta-llama/llama-3-70b-instruct": 4096,
Expand Down Expand Up @@ -401,8 +407,22 @@
# LiteLLMAIHandler routes OpenRouter-prefixed forms through extra_body.reasoning.
"gemini-2.5-pro",
"gemini-2.5-flash",
# Register each published Grok id separately so provider-prefixed forms match
# and the allowlist below can clamp model-specific reasoning levels.
"grok-4.5",
"grok-4.5-latest",
"grok-build-latest",
"grok-4.6",
]

# Clamp OpenAI-only levels for always-on Grok reasoning; allow xhigh on 4.6+.
GROK_REASONING_EFFORT_LEVELS = {
"grok-4.5": {"low", "medium", "high"},
"grok-4.5-latest": {"low", "medium", "high"},
"grok-build-latest": {"low", "medium", "high"},
"grok-4.6": {"low", "medium", "high", "xhigh"},
}

# Claude models that support "extended thinking" through the manual
# thinking={"type": "enabled", "budget_tokens": ...} request built by
# LiteLLMAIHandler._configure_claude_extended_thinking(). Only models that
Expand Down
63 changes: 60 additions & 3 deletions pr_agent/algo/ai_handlers/litellm_ai_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

from pr_agent.algo import (
CLAUDE_EXTENDED_THINKING_MODELS,
GROK_REASONING_EFFORT_LEVELS,
NO_SUPPORT_TEMPERATURE_MODELS,
STREAMING_REQUIRED_MODELS,
SUPPORT_REASONING_EFFORT_MODELS,
Expand Down Expand Up @@ -449,6 +450,33 @@ def _has_priceable_usage(usage) -> bool:
)
)

@staticmethod
def _grok_reasoning_levels_for(model: str) -> set[str] | None:
"""Return the reasoning-effort levels accepted by a registered Grok model."""
normalized_model = model.rsplit(":", 1)[0] if model.startswith("openrouter/") else model
return next(
(
levels
for grok_id, levels in GROK_REASONING_EFFORT_LEVELS.items()
if normalized_model == grok_id or normalized_model.endswith("/" + grok_id)
),
None,
)

@classmethod
def _clamp_grok_reasoning_effort(cls, model: str, reasoning_effort: str) -> str:
"""Clamp a configured reasoning effort to the closest supported Grok level."""
grok_levels = cls._grok_reasoning_levels_for(model)
if not grok_levels or reasoning_effort in grok_levels:
return reasoning_effort
try:
ReasoningEffort(reasoning_effort)
except (ValueError, TypeError):
return reasoning_effort
if reasoning_effort in ("max", "xhigh"):
return "xhigh" if "xhigh" in grok_levels else "high"
return "low"

def _configure_claude_extended_thinking(self, model: str, kwargs: dict) -> dict:
"""
Configure Claude extended thinking parameters if applicable.
Expand Down Expand Up @@ -753,6 +781,9 @@ async def chat_completion(self, model: str, system: str, user: str, temperature:
if 'temperature' in kwargs:
del kwargs['temperature']

custom_llm_provider = str(
getattr(get_settings().litellm, "custom_llm_provider", "") or ""
).strip().lower()
openrouter_reasoning_effort = None
reasoning_model = model.rsplit(":", 1)[0] if model.startswith("openrouter/") else model
# Add reasoning_effort if model supports it. Match the bare model
Expand All @@ -777,6 +808,14 @@ async def chat_completion(self, model: str, system: str, user: str, temperature:
f"Using default '{reasoning_effort}'. Valid values: {[e.value for e in ReasoningEffort]}"
)

clamped_effort = self._clamp_grok_reasoning_effort(model, reasoning_effort)
if clamped_effort != reasoning_effort:
get_logger().info(
f"Grok model {model} does not support reasoning_effort='{reasoning_effort}'; "
f"using '{clamped_effort}' instead."
)
reasoning_effort = clamped_effort

if model.startswith("openrouter/"):
# LiteLLM 1.98.0 rejects top-level reasoning_effort for some
# OpenRouter model IDs it does not mark as reasoning-capable;
Expand All @@ -785,6 +824,18 @@ async def chat_completion(self, model: str, system: str, user: str, temperature:
else:
get_logger().info(f"Adding reasoning_effort with value {reasoning_effort} to model {model}.")
kwargs["reasoning_effort"] = reasoning_effort
if self._grok_reasoning_levels_for(model):
try:
supported_params = litellm.get_supported_openai_params(
model=model,
custom_llm_provider=custom_llm_provider or None,
) or []
except Exception:
supported_params = []
# LiteLLM 1.98.0 omits reasoning_effort for grok-build-latest
# and OpenAI-compatible gateway-prefixed Grok IDs.
if "reasoning_effort" not in supported_params:
kwargs["allowed_openai_params"] = ["reasoning_effort"]

# https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking
if (model in self.claude_extended_thinking_models) and get_settings().config.get("enable_claude_extended_thinking", False):
Expand Down Expand Up @@ -946,6 +997,15 @@ def _as_int(value):
elif reasoning_max_tokens <= 0:
effective_reasoning_effort = openrouter_reasoning_effort or ""

if effective_reasoning_effort:
clamped_effort = self._clamp_grok_reasoning_effort(model, effective_reasoning_effort)
if clamped_effort != effective_reasoning_effort:
get_logger().info(
f"Grok model {model} does not support reasoning_effort="
f"'{effective_reasoning_effort}'; using '{clamped_effort}' instead."
)
effective_reasoning_effort = clamped_effort

# Preserve explicit disablement; otherwise keep effort and
# max_tokens mutually exclusive by preferring the token budget.
if effective_reasoning_effort == "none":
Expand Down Expand Up @@ -1007,9 +1067,6 @@ def _as_int(value):

# Optional fixed provider override, so a raw hosted model id reaches the
# provider unchanged instead of being rewritten by LiteLLM's prefix inference.
custom_llm_provider = str(
getattr(get_settings().litellm, "custom_llm_provider", "") or ""
).strip().lower()
if custom_llm_provider:
kwargs["custom_llm_provider"] = custom_llm_provider

Expand Down
4 changes: 3 additions & 1 deletion pr_agent/settings/configuration.toml
Original file line number Diff line number Diff line change
Expand Up @@ -420,7 +420,9 @@ allow_fallbacks = true # when provider_order is set, allow routing beyond the li
# Model-specific support varies; mandatory reasoning models reject "none".
reasoning_effort = ""
# A positive value overrides global effort and non-none OpenRouter-specific efforts.
# Explicit openrouter.reasoning_effort = "none" keeps reasoning disabled.
# Explicit openrouter.reasoning_effort = "none" keeps reasoning disabled, except on
# Grok 4.5/4.6: there "none" is clamped to the lowest supported
# effort first, so a positive budget wins over it.
# Some providers require max_tokens to be greater than the reasoning budget.
Comment thread
PeterDaveHello marked this conversation as resolved.
reasoning_max_tokens = 0
max_tokens = 0 # hard cap on completion tokens for the request; 0 = unset
Expand Down
20 changes: 20 additions & 0 deletions tests/unittest/test_get_max_tokens.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,26 @@ def test_bedrock_mantle_grok_4_3_model_max_tokens(self, monkeypatch):

assert get_max_tokens("bedrock_mantle/xai.grok-4.3") == 1000000

@pytest.mark.parametrize("model", [
"xai/grok-4.5",
"xai/grok-4.5-latest",
"xai/grok-build-latest",
"xai/grok-4.6",
"openrouter/x-ai/grok-4.5",
"openrouter/x-ai/grok-4.6",
])
def test_xai_and_openrouter_grok_4_5_and_4_6_models_max_tokens(self, monkeypatch, model):
fake_settings = type("", (), {
"config": type("", (), {
"custom_model_max_tokens": 0,
"max_model_tokens": 0,
})()
})()

monkeypatch.setattr(utils, "get_settings", lambda: fake_settings)

assert get_max_tokens(model) == 500000

@pytest.mark.parametrize(
"model",
[
Expand Down
Loading
Loading