Skip to content

fix: Improve model compatibility in parsing and logging #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
abinashkarki wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix: Improve model compatibility in parsing and logging #7

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6a38945
fix: Handle special tokens around FINAL statements in parsing
abinashkarki Jan 4, 2026
dd6872f
Merge branch 'main' into fix/model-compatibility
alexzhang13 Jan 11, 2026
053c76b
fix: Stricter FINAL pattern to reject CoT false positives
abinashkarki Jan 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions rlm/utils/parsing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,15 +33,27 @@ def find_final_answer(text: str, environment: "BaseEnv | None" = None) -> str |
If FINAL_VAR is found and an environment is provided, executes code to retrieve the variable value.
Returns None if neither pattern is found.

Only accepts FINAL/FINAL_VAR in these positions to avoid false positives from CoT reasoning:
1. At the start of a line (with optional leading whitespace)
2. Immediately after special tokens like <|begin_of_box|>

This rejects cases like "I will then do FINAL(...)" where the model is explaining its plan.

Args:
text: The response text to parse
environment: Optional environment to execute code for FINAL_VAR retrieval

Returns:
The final answer string, or None if no final answer pattern is found
"""
# Check for FINAL_VAR pattern first - must be at start of line
final_var_pattern = r"^\s*FINAL_VAR\((.*?)\)"
# Pattern explanation:
# (?:^[ \t]*|(?<=\n)[ \t]*|<\|[^|]+\|>\s*) matches:
# - Start of string with optional spaces/tabs: ^[ \t]*
# - After newline with optional spaces/tabs: (?<=\n)[ \t]*
# - After special tokens like <|...|>: <\|[^|]+\|>\s*

# Check for FINAL_VAR pattern first (more specific)
final_var_pattern = r"(?:^[ \t]*|(?<=\n)[ \t]*|<\|[^|]+\|>\s*)FINAL_VAR\((.*?)\)"
match = re.search(final_var_pattern, text, re.MULTILINE | re.DOTALL)
if match:
variable_name = match.group(1).strip().strip('"').strip("'")
Expand All @@ -53,8 +65,8 @@ def find_final_answer(text: str, environment: "BaseEnv | None" = None) -> str |
return final_answer
return None

# Check for FINAL pattern - must be at start of line
final_pattern = r"^\s*FINAL\((.*?)\)"
# Check for FINAL pattern
final_pattern = r"(?:^[ \t]*|(?<=\n)[ \t]*|<\|[^|]+\|>\s*)FINAL\((.*?)\)"
match = re.search(final_pattern, text, re.MULTILINE | re.DOTALL)
if match:
return match.group(1).strip()
Expand Down