Skip to content

Commit d04001a

Browse files
derekhigginsclaude
andcommitted
fix(tests): validate provider types exist in backward compat test
The backward compatibility test previously only validated YAML schema structure but did not verify that referenced provider types actually exist in the registry. This would have allowed breaking changes like renaming "meta-reference" to "builtin" (236a230) to pass without detection(although the PR was correctly marked as a breaking change). This commit enhances the test to use `validate_and_prepare_providers()`, the same validation function the stack uses at runtime, which now checks: - Provider types exist in registry - No conflicts with auto-routed APIs - Provider deprecation warnings/errors Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Derek Higgins <derekh@redhat.com>
1 parent 839b48a commit d04001a

1 file changed

Lines changed: 18 additions & 1 deletion

File tree

tests/backward_compat/test_run_config.py

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@
1818
import yaml
1919

2020
from llama_stack.core.datatypes import StackConfig
21+
from llama_stack.core.distribution import builtin_automatically_routed_apis, get_provider_registry
22+
from llama_stack.core.resolver import validate_and_prepare_providers
2123
from llama_stack.core.stack import replace_env_vars
2224

2325

@@ -58,4 +60,19 @@ def test_load_run_config(config_file):
5860
except Exception:
5961
pass
6062

61-
StackConfig.model_validate(config_data)
63+
# Validate schema
64+
config = StackConfig.model_validate(config_data)
65+
66+
# Validate providers using the same validation the stack uses at runtime
67+
# This validates:
68+
# - Provider types exist in registry
69+
# - No conflicts with auto-routed APIs
70+
# - Provider deprecation warnings/errors
71+
provider_registry = get_provider_registry()
72+
routing_table_apis = {x.routing_table_api for x in builtin_automatically_routed_apis()}
73+
router_apis = {x.router_api for x in builtin_automatically_routed_apis()}
74+
75+
try:
76+
validate_and_prepare_providers(config, provider_registry, routing_table_apis, router_apis)
77+
except ValueError as e:
78+
pytest.fail(f"Provider validation failed: {str(e)}")

0 commit comments

Comments
 (0)