Fix vLLM harness provider kind precedence#236
Open
abhinav-bellapu wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes the provider-kind mismatch described in #123 for Beaker launches that use a harness preset such as
defaultwith a model preset whose provider kind isvllm.The root cause was that
HarnessConfig.merge_provider()treated harness provider fields as explicit only when they differed from the defaultProviderConfig(). Sincevllm_serveris also the default provider kind, a named harness preset that explicitly selectsvllm_servercould still lose that kind when merged with a model preset that selectsvllm. Beaker job assembly would then install vLLM for the isolatedvllm_serverpath, while runtime config could select the in-processvllmprovider.This change lets callers preserve the harness provider kind when they know it came from an explicit harness preset, config, or override. Empty/default harness configs still allow the model preset provider kind to win, preserving the existing non-harness behavior.
Reviewer notes:
Closes #123
Type of Change
Testing
pytest tests/ --ignore=tests/integration/)Local checks run:
uv run --no-group vllm ruff format --check src/olmo_eval/harness/config.py src/olmo_eval/cli/run/config.py tests/core/harness/test_harness.py tests/cli/run/__init__.py tests/cli/run/test_config.py tests/cli/beaker/test_launch.pyuv run --no-group vllm ruff check src/olmo_eval/harness/config.py src/olmo_eval/cli/run/config.py tests/core/harness/test_harness.py tests/cli/run/__init__.py tests/cli/run/test_config.py tests/cli/beaker/test_launch.pyuv run --no-group vllm pytest tests/core/harness/test_harness.py tests/cli/run/test_config.py tests/cli/beaker/test_launch.py::TestHarnessOverridesProviderDependenciesuv run --no-group vllm pytest tests/cli tests/core/harnessChecklist