Repository navigation
Introduce category definitions for category metadata - #933
Draft
Sun Haoran (haoranpb) wants to merge 1 commit into
Draft
Sun Haoran (haoranpb) wants to merge 1 commit into
Sun Haoran (haoranpb) wants to merge 1 commit into
Conversation
Sun Haoran (haoranpb)
added this pull request to stack #960
October 8, 2026 12:17
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| EXTERNAL_COMMAND = "external-command" | ||
|
|
||
|
|
||
| CopilotModelName = Literal[ |
| from collections.abc import Sequence | ||
| from dataclasses import dataclass | ||
| from pathlib import Path | ||
| from typing import Generic, Literal, TypeVar |
| __all__ = ["CategoryDefinition", "category_definition"] | ||
|
|
||
|
|
||
| def category_definition(category: EvaluationCategory) -> CategoryDefinition[BaseDatasetEntry]: |
Goal 2 step 2, following the hybrid ownership model: each category gets a package under bcbench.categories, starting with a typed definition.py. - CategoryDefinition holds what was static metadata on EvaluationCategory: dataset file and entry type (requires_repo follows from it), bc-eval evaluators and core score, runner label, container requirement, and whether the agent may see the BC container credentials. - category_definition() maps a category to its definition with an exhaustive match; the CLI and commands use it, and the corresponding EvaluationCategory properties are removed. - Runtime code receives inputs instead of looking them up: the Copilot and Claude agents take pass_bc_credentials, the bc-eval export takes the loaded dataset entries, and the red-team target takes its entry. - CopilotModelName moves to bcbench.types so config.py no longer imports the CLI options module. Pipelines, result types, judge models and prompts still hang off EvaluationCategory and move in the next steps. CATEGORIES.md and the Copilot instructions describe the new layout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sun Haoran (haoranpb)
force-pushed
the
app/stack-02-category-definitions
branch
from
October 8, 2026 13:13
6634519 to
239bdda
Compare
Contributor
Code Coverage OverviewLanguages: Python Python / code-coverage/pytestThe overall line coverage in commit 239bdda in the Show a line coverage summary of the most impacted files.
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal 2 step 2, stacked on #932. Starts replacing the
EvaluationCategorylookup table with category-owned definitions (hybrid layout: category-specific code underbcbench/categories/<name>/, shared pieces stay common).bcbench/categories/: a typedCategoryDefinitionand one package per category with itsdefinition.py(dataset file, entry type, evaluators, core score, runner, container requirement, credential policy).category_definition()dispatches with an exhaustivematch.EvaluationCategoryproperties are removed; the CLI and commands use the definitions.pass_bc_credentials, the bc-eval export takes loaded entries, the red-team target takes its entry.Pipelines, result types, judge models and prompts move in the next steps. No behaviour change; CLI outputs verified against the old values.