Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

max_iterations Not Limiting Agent Execution Steps #6857

Closed
Hambaobao opened this issue Feb 20, 2025 · 5 comments
Closed

max_iterations Not Limiting Agent Execution Steps #6857

Hambaobao opened this issue Feb 20, 2025 · 5 comments
Labels
bug Something isn't working configuration Related to configuring OpenHands settings openhands Have Openhands attempt to fix the issue (via GH App)

Comments

@Hambaobao
Copy link

Description

I attempted to limit the agent’s execution steps by setting max_iterations = 1 in the config file. However, despite this setting, the agent continues executing up to step 6.

Config

[core]
default_agent = "CodeActAgent"
run_as_openhands = false
runtime = "docker"
debug = true
file_store = "memory"

workspace_mount_path = "/tmp/tmp5reox87y"
workspace_mount_path_in_sandbox = "/workspace"

max_iterations = 1
save_trajectory_path = "outputs/psf____requests/trajectory.json"

[llm]
model = "openai/xxx"
api_key = "sk-b"
base_url = "x"
embedding_model = ""
temperature = 0.0
top_p = 1.0

[sandbox]
# runtime container image
runtime_container_image = "unitflow:build-env-base"

Log

==============
[Agent Controller default] LEVEL 0 LOCAL STEP 6 GLOBAL STEP 6

Building environments... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0/6479 0:03:46 -:--:--23:19:32 - openhands:INFO: agent_controller.py:729
ACTION
[Agent Controller default] AgentFinishAction(outputs={}, thought="Great! The environment setup is now complete and successful. Here's a summary of what we achieved:\n\n1. Successfully installed the package in editable mode (`pip install -e .`)\n2. Successfully installed all development dependencies from `requirements-dev.txt`\n3. Test results show:\n   - 590 tests passed\n   - 15 tests skipped\n   - 1 test xfailed (expected failure)\n   - 18 warnings (mostly deprecation warnings for future Requests 3.0.0)\n   - All tests completed in 78.32 seconds\n   - Most importantly, there are no dependency-related errors\n\nThe environment is now properly set up for development. The warnings are related to:\n- Future deprecations in Requests 3.0.0 for non-string usernames/passwords\n- Some insecure HTTPS requests in tests\n- Deprecation of `get_encodings_from_content` in future Requests 3.0.0\n\nThese warnings don't affect the functionality of the development environment.", action='finish')
23:19:32 - openhands:INFO: agent_controller.py:451 - [Agent Controller default] Setting agent(CodeActAgent) state from AgentState.RUNNING to AgentState.FINISHED
23:19:32 - openhands:INFO: agent_controller.py:363
OBSERVATION
[Agent Controller default] NullObservation(content='', observation='null')
23:19:32 - openhands:INFO: agent_controller.py:363
OBSERVATION
[Agent Controller default] AgentStateChangedObservation(content='', agent_state=<AgentState.FINISHED: 'finished'>, observation='agent_state_changed')
@mamoodi mamoodi added bug Something isn't working configuration Related to configuring OpenHands settings labels Feb 20, 2025
@mamoodi
Copy link
Collaborator

mamoodi commented Feb 20, 2025

Just want to check that you are running in development mode right?

@Hambaobao
Copy link
Author

Just want to check that you are running in development mode right?

The version I am using is 0.23.0. I downloaded the release code and installed openhands locally using poetry install and pip install -e .. If by "development mode" you mean this setup, then yes, I am running it in development mode.

@neubig neubig added the openhands Have Openhands attempt to fix the issue (via GH App) label Feb 20, 2025
@openhands-ai OpenHands AI
Copy link

openhands-ai bot commented Feb 20, 2025

I'm working on a fix! @neubig can track my progress at all-hands.dev

@neubig neubig mentioned this issue Feb 20, 2025
Closed
neubig pushed a commit that referenced this issue Feb 20, 2025
@openhands-agent
Fix max_iterations not being enforced in headless mode
d38f23a 
The max_iterations setting in the config file was not being enforced when running in headless mode. This was because we were only checking the iteration limit in the _step method, but not in the _handle_observation method. This meant that when an agent action triggered a command execution, the command output observation would trigger another step even if we had hit the iteration limit.

Changes:
- Modified _handle_observation to check iteration limit before processing observations
- Added test case test_iteration_limit_in_headless_mode_with_agent_steps that verifies this behavior

Fixes #6857
@enyst
Copy link
Collaborator

enyst commented Feb 21, 2025 •
edited
Loading

Ah I see. In evals, we don't actually read the config.toml file, except for LLMs configurations. The rest are manually made, for example here:

OpenHands/evaluation/benchmarks/swe_bench/run_infer.py

Line 136 in 22c5ad8

config = AppConfig(

As you can see, we create an instance with those values, instead of loading it from the file. This fools me all the time... 😅

Cc: @neubig

@neubig
Copy link
Contributor

neubig commented Feb 22, 2025

Thanks @enyst . I think long story short you'll need to pass in the configuration through command line arguments, not config.toml. So add --max-iterations X when you call run_infer.py.

This is yet another instance of #3220 , which I'd really like to fix but it's a big effort so we haven't gotten to it.

Anyway, I'll close this for now since there should be a clear work-around.

@neubig neubig closed this as completed Feb 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working configuration Related to configuring OpenHands settings openhands Have Openhands attempt to fix the issue (via GH App)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix max_iterations not being enforced in headless mode All-Hands-AI/OpenHands
4 participants
@neubig @enyst @Hambaobao @mamoodi