Skip to content

Fix vLLM harness provider kind precedence#236

Open
abhinav-bellapu wants to merge 1 commit into
allenai:mainfrom
abhinav-bellapu:fix-harness-provider-kind
Open

Fix vLLM harness provider kind precedence#236
abhinav-bellapu wants to merge 1 commit into
allenai:mainfrom
abhinav-bellapu:fix-harness-provider-kind

Conversation

@abhinav-bellapu

@abhinav-bellapu abhinav-bellapu commented Jul 1, 2026

Copy link
Copy Markdown

Description

Fixes the provider-kind mismatch described in #123 for Beaker launches that use a harness preset such as default with a model preset whose provider kind is vllm.

The root cause was that HarnessConfig.merge_provider() treated harness provider fields as explicit only when they differed from the default ProviderConfig(). Since vllm_server is also the default provider kind, a named harness preset that explicitly selects vllm_server could still lose that kind when merged with a model preset that selects vllm. Beaker job assembly would then install vLLM for the isolated vllm_server path, while runtime config could select the in-process vllm provider.

This change lets callers preserve the harness provider kind when they know it came from an explicit harness preset, config, or override. Empty/default harness configs still allow the model preset provider kind to win, preserving the existing non-harness behavior.

Reviewer notes:

  • The fix is applied at the config-merge boundary so runtime provider selection and Beaker dependency assembly share the same provider kind.
  • The new preservation behavior is opt-in for call sites that have already materialized an explicit harness config; plain model-provider merges keep their previous precedence.
  • Coverage includes the merge helper, CLI config resolution, and Beaker dependency assembly path that originally exposed the mismatch.
  • No user-facing docs were changed because this corrects preset precedence behavior without adding new CLI/config syntax.

Closes #123

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)

Testing

  • Unit tests pass locally (pytest tests/ --ignore=tests/integration/)
  • Integration tests pass (if applicable)
  • New tests added for new functionality

Local checks run:

  • uv run --no-group vllm ruff format --check src/olmo_eval/harness/config.py src/olmo_eval/cli/run/config.py tests/core/harness/test_harness.py tests/cli/run/__init__.py tests/cli/run/test_config.py tests/cli/beaker/test_launch.py
  • uv run --no-group vllm ruff check src/olmo_eval/harness/config.py src/olmo_eval/cli/run/config.py tests/core/harness/test_harness.py tests/cli/run/__init__.py tests/cli/run/test_config.py tests/cli/beaker/test_launch.py
  • uv run --no-group vllm pytest tests/core/harness/test_harness.py tests/cli/run/test_config.py tests/cli/beaker/test_launch.py::TestHarnessOverridesProviderDependencies
  • uv run --no-group vllm pytest tests/cli tests/core/harness

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my code
  • I have added/updated documentation as needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

@abhinav-bellapu abhinav-bellapu changed the title Fix harness provider kind precedence Fix vLLM harness provider kind precedence Jul 1, 2026
@abhinav-bellapu abhinav-bellapu marked this pull request as ready for review July 1, 2026 04:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] vLLM venv mismatch between harness preset and model preset

1 participant