Introduce category definitions for category metadata

.github/copilot-instructions.md

@@ -34,7 +34,7 @@ BC-Bench is category-based and designed to grow over time. It currently has two
 
 Preserve one-way dependency flow from orchestration toward lower-level abstractions. Keep domain and result models independent of runtime code. CLI commands are composition roots: they select concrete agents and inject them into evaluation pipelines through `AgentRunner`; pipelines must not select agent implementations.
 
-`bcbench.types` is the central category registry. Extend `EvaluationCategory` for category-owned mappings such as datasets, pipelines, results, and scoring behavior instead of duplicating those decisions elsewhere. Keep imports following the existing direction and avoid circular dependencies.
+Each evaluation category owns a package under `bcbench.categories` (a typed `CategoryDefinition` plus its category-specific code); code shared by several categories stays in the common modules. The CLI selects categories with `category_definition()`; runtime code receives what it needs explicitly instead of looking it up on `EvaluationCategory`. Keep imports following the existing direction and avoid circular dependencies.
 
 Logging setup lives in bcbench-core (`bcbench_core.logs.setup_logging`) so applications reuse it rather than copy it; the CLI calls it once, and modules only use `logging.getLogger(__name__)`.
 

CATEGORIES.md

@@ -2,23 +2,29 @@
 
 BC-Bench is **category-based**. A category is a distinct evaluation scenario: `bug-fix` asks an agent to patch buggy code, `test-generation` asks it to write reproduction tests, `code-review` asks it to flag issues in a diff, `nl2al` asks it to turn a natural-language spec into AL code, `extensibility-request-advisor` asks it to classify and assess a fully specified extensibility need and draft a GitHub issue, `extensibility-request-implement` asks it to implement an approved extensibility request (e.g. add an integration event) in an existing repo, and `extensibility-request-triage` asks it to triage an extensibility request (emit managed labels, an advisory comment, and an open/closed decision).
 
-Categories also differ in how they're scored and run. `bug-fix` and `test-generation` are execution-based: they build and run AL code, so they need a BC container. `code-review`, `nl2al`, `extensibility-request-advisor`, `extensibility-request-implement`, and `extensibility-request-triage` all leverage LLM-as-a-judge: `code-review` scores precision/recall/F1 of flagged issues against expected findings (an LLM judge only matches comments), and the other judge-based categories have an LLM grade the agent output against an LMChecklist. Advisor entries use an offline, single-shot proxy for the production skill's interactive flow: the prompt supplies the complete scenario, GitHub submission is disabled, and the agent persists classification, feasibility analysis, alternatives, and the final issue draft to `advisor_result.json`. The `EvaluationCategory` properties (`requires_container`, `runner`, `evaluators`, `core_score`) capture these differences for the workflows.
+Categories also differ in how they're scored and run. `bug-fix` and `test-generation` are execution-based: they build and run AL code, so they need a BC container. `code-review`, `nl2al`, `extensibility-request-advisor`, `extensibility-request-implement`, and `extensibility-request-triage` all leverage LLM-as-a-judge: `code-review` scores precision/recall/F1 of flagged issues against expected findings (an LLM judge only matches comments), and the other judge-based categories have an LLM grade the agent output against an LMChecklist. Advisor entries use an offline, single-shot proxy for the production skill's interactive flow: the prompt supplies the complete scenario, GitHub submission is disabled, and the agent persists classification, feasibility analysis, alternatives, and the final issue draft to `advisor_result.json`. Each category's definition (`requires_container`, `runner`, `evaluators`, `core_score`) captures these differences for the workflows.
 
 Categories may share a dataset (`bug-fix` and `test-generation` do today), but a new category should generally have its own: dataset schema, entry type, result type, pipeline, etc.
 
 This doc is a map; the source files and their comments are the source of truth. To experiment with agent setup on existing categories, see [EXPERIMENT.md](EXPERIMENT.md).
 
 ## Architecture
 
-Start with `EvaluationCategory` in [src/bcbench/types.py](src/bcbench/types.py). It is the category registry. Each enum value maps to the pieces the rest of the CLI and workflows consume:
+`EvaluationCategory` in [src/bcbench/types.py](src/bcbench/types.py) names the categories. Each category has a package under [src/bcbench/categories/](src/bcbench/categories/) that owns its behaviour; code shared by several categories (base classes, shared result families, shared datasets) stays in the common modules. The CLI looks up a category with `category_definition()` in [src/bcbench/categories/__init__.py](src/bcbench/categories/__init__.py).
+
+A category's `definition.py` declares ([src/bcbench/categories/definition.py](src/bcbench/categories/definition.py)):
+
+- `dataset_file` / `entry_type` — the dataset file for raw tasks and the typed Python model for one dataset row (aka one task). `requires_repo` follows from the entry type.
+- `evaluators` / `core_score` — the bc-eval evaluator list and headline score, emitted to workflows by [src/bcbench/commands/category.py](src/bcbench/commands/category.py).
+- `requires_container` / `runner` — whether the category needs a BC container, and which runner evaluates it.
+- `pass_bc_credentials` — whether the agent may see the BC container credentials.
+
+Still mapped from `EvaluationCategory` while the redesign continues:
 
-- `dataset_path` — the dataset file for raw tasks.
-- `entry_class` — the typed Python model for one dataset row (aka one task).
 - `result_class` — the recorded outcome for one evaluated task.
 - `summary_class` / `aggregate_class` — the aggregate views used by result summaries and leaderboards.
 - `pipeline` — the category-specific setup, agent run, and evaluation behavior.
-- `evaluators` / `core_score` — the bc-eval evaluator list and headline score, emitted to workflows by [src/bcbench/commands/category.py](src/bcbench/commands/category.py).
-- `requires_container` / `requires_repo` / `runner` — whether the category needs a BC container, whether it needs the dataset repository cloned, and which runner evaluates it.
+- `judge_model` — the pinned LLM judge for judge-scored categories.
 - Prompt template — the category-specific prompt in [src/bcbench/agent/shared/config.yaml](src/bcbench/agent/shared/config.yaml), loaded by [src/bcbench/agent/shared/prompt.py](src/bcbench/agent/shared/prompt.py).
 
 Keep dataset entry classes and result classes focused on typed data. Put category-specific behavior in the pipeline.
@@ -27,7 +33,7 @@ Keep dataset entry classes and result classes focused on typed data. Put categor
 
 Use the existing implementations as examples: `bug-fix` and `test-generation` for execution-based categories, `code-review` and `nl2al` for judge-based ones.
 
-1. Add the enum value and mappings in [src/bcbench/types.py](src/bcbench/types.py).
+1. Add the enum value and remaining mappings in [src/bcbench/types.py](src/bcbench/types.py), and a category package with `definition.py` under [src/bcbench/categories/](src/bcbench/categories/), wired into `category_definition()`.
 2. Add the category dataset JSONL and entry class in [src/bcbench/dataset/dataset_entry.py](src/bcbench/dataset/dataset_entry.py).
 3. Add a result class under [src/bcbench/results/](src/bcbench/results/) and map it from `EvaluationCategory.result_class`.
 4. Add a pipeline under [src/bcbench/evaluate/](src/bcbench/evaluate/).

src/bcbench/agent/claude/agent.py

@@ -34,6 +34,7 @@ def run_claude_code(
     category: EvaluationCategory,
     repo_path: Path,
     output_dir: Path,
+    pass_bc_credentials: bool,
     runtime: AgentRuntimeConfig | None = None,
     timeout: int = 60 * 60,
 ) -> tuple[AgentMetrics | None, ExperimentConfiguration]:
@@ -132,7 +133,7 @@ def run_claude_code(
                     "MCP_TIMEOUT": "180000",
                     "MCP_TOOL_TIMEOUT": "180000",
                 },
-                pass_bc_credentials=category.pass_on_bc_container_credentials,
+                pass_bc_credentials=pass_bc_credentials,
             ),
             timeout=timeout,
             check=True,

src/bcbench/agent/copilot/agent.py

@@ -28,6 +28,7 @@ def run_copilot_agent(
     category: EvaluationCategory,
     repo_path: Path,
     output_dir: Path,
+    pass_bc_credentials: bool,
     runtime: AgentRuntimeConfig | None = None,
     timeout: int = 60 * 60,
 ) -> tuple[AgentMetrics | None, ExperimentConfiguration]:
@@ -93,7 +94,7 @@ def run_copilot_agent(
             model=model,
             work_dir=repo_path,
             timeout=timeout,
-            env=agent_subprocess_env(pass_bc_credentials=category.pass_on_bc_container_credentials),
+            env=agent_subprocess_env(pass_bc_credentials=pass_bc_credentials),
             options=options,
         )
         logger.info(f"Copilot CLI run complete for: {entry.instance_id}")

src/bcbench/categories/__init__.py

@@ -0,0 +1,39 @@
+"""Evaluation categories: each package under bcbench.categories owns one category's definition and behaviour."""
+
+from typing import assert_never
+
+from bcbench.categories.bug_fix.definition import DEFINITION as BUG_FIX
+from bcbench.categories.code_review.definition import DEFINITION as CODE_REVIEW
+from bcbench.categories.data_query.definition import DEFINITION as DATA_QUERY
+from bcbench.categories.definition import CategoryDefinition
+from bcbench.categories.ext_request_advisor.definition import DEFINITION as EXT_REQUEST_ADVISOR
+from bcbench.categories.ext_request_implement.definition import DEFINITION as EXT_REQUEST_IMPLEMENT
+from bcbench.categories.ext_request_triage.definition import DEFINITION as EXT_REQUEST_TRIAGE
+from bcbench.categories.nl2al.definition import DEFINITION as NL2AL
+from bcbench.categories.test_generation.definition import DEFINITION as TEST_GENERATION
+from bcbench.dataset import BaseDatasetEntry
+from bcbench.types import EvaluationCategory
+
+__all__ = ["CategoryDefinition", "category_definition"]
+
+
+def category_definition(category: EvaluationCategory) -> CategoryDefinition[BaseDatasetEntry]:
+    match category:
+        case EvaluationCategory.BUG_FIX:
+            return BUG_FIX
+        case EvaluationCategory.TEST_GENERATION:
+            return TEST_GENERATION
+        case EvaluationCategory.CODE_REVIEW:
+            return CODE_REVIEW
+        case EvaluationCategory.NL2AL:
+            return NL2AL
+        case EvaluationCategory.DATA_QUERY:
+            return DATA_QUERY
+        case EvaluationCategory.EXT_REQUEST_ADVISOR:
+            return EXT_REQUEST_ADVISOR
+        case EvaluationCategory.EXT_REQUEST_IMPLEMENT:
+            return EXT_REQUEST_IMPLEMENT
+        case EvaluationCategory.EXT_REQUEST_TRIAGE:
+            return EXT_REQUEST_TRIAGE
+        case _:
+            assert_never(category)

src/bcbench/categories/bug_fix/definition.py

@@ -0,0 +1,14 @@
+from bcbench.categories.definition import CategoryDefinition
+from bcbench.dataset import BugFixEntry
+from bcbench.types import EvaluationCategory
+
+DEFINITION = CategoryDefinition(
+    category=EvaluationCategory.BUG_FIX,
+    dataset_file="bcbench.jsonl",
+    entry_type=BugFixEntry,
+    evaluators=("resolution_rate", "build_rate"),
+    core_score="ResolutionRate",
+    runner="GitHub-BCBench",
+    requires_container=True,
+    pass_bc_credentials=True,
+)

src/bcbench/categories/code_review/definition.py

@@ -0,0 +1,14 @@
+from bcbench.categories.definition import CategoryDefinition
+from bcbench.dataset import CodeReviewEntry
+from bcbench.types import EvaluationCategory
+
+DEFINITION = CategoryDefinition(
+    category=EvaluationCategory.CODE_REVIEW,
+    dataset_file="codereview.jsonl",
+    entry_type=CodeReviewEntry,
+    evaluators=("precision_score", "recall_score", "f1_score", "valid_review_output"),
+    core_score="F1Score",
+    runner="ubuntu-latest",
+    requires_container=False,
+    pass_bc_credentials=True,
+)

src/bcbench/categories/data_query/definition.py

@@ -0,0 +1,14 @@
+from bcbench.categories.definition import CategoryDefinition
+from bcbench.dataset import DataQueryEntry
+from bcbench.types import EvaluationCategory
+
+DEFINITION = CategoryDefinition(
+    category=EvaluationCategory.DATA_QUERY,
+    dataset_file="dataquery.jsonl",
+    entry_type=DataQueryEntry,
+    evaluators=("resolution_rate", "build_rate"),
+    core_score="ResolutionRate",
+    runner="GitHub-BCBench",
+    requires_container=True,
+    pass_bc_credentials=False,
+)

src/bcbench/categories/definition.py

@@ -0,0 +1,41 @@
+"""The contract every evaluation category fulfils."""
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Generic, Literal, TypeVar
+
+from bcbench.dataset import BaseDatasetEntry, RepoGroundedEntry
+from bcbench.types import EvaluationCategory
+
+# Covariant so a definition for a specific entry type is usable wherever any definition is expected
+E_co = TypeVar("E_co", bound=BaseDatasetEntry, covariant=True)
+
+# GitHub Actions runner labels; only categories that build BaseApp need the self-hosted runners
+type Runner = Literal["GitHub-BCBench", "ubuntu-latest", "windows-latest"]
+
+
+@dataclass(frozen=True)
+class CategoryDefinition(Generic[E_co]):  # noqa: UP046 - PEP 695 syntax cannot declare covariance explicitly
+    category: EvaluationCategory
+    dataset_file: str
+    entry_type: type[E_co]
+    # bc-eval evaluators (evaluator/scores.py) uploaded for this category, and the one reported as CoreScore
+    evaluators: tuple[str, ...]
+    core_score: str
+    runner: Runner
+    # Whether evaluating builds/runs AL code and therefore needs a BC container
+    requires_container: bool
+    # Development scenarios (e.g. bug-fix) get direct container access; production-like ones (e.g. data-query) must
+    # use the exposed access methods, such as BC MCP, so the agent never sees the container credentials
+    pass_bc_credentials: bool
+
+    @property
+    def requires_repo(self) -> bool:
+        return issubclass(self.entry_type, RepoGroundedEntry)
+
+    def dataset_path(self, dataset_dir: Path) -> Path:
+        return dataset_dir / self.dataset_file
+
+    def load_entries(self, dataset_dir: Path, entry_id: str | None = None) -> Sequence[E_co]:
+        return self.entry_type.load(self.dataset_path(dataset_dir), entry_id=entry_id)

src/bcbench/categories/ext_request_advisor/definition.py

@@ -0,0 +1,14 @@
+from bcbench.categories.definition import CategoryDefinition
+from bcbench.dataset import ExtRequestAdvisorEntry
+from bcbench.types import EvaluationCategory
+
+DEFINITION = CategoryDefinition(
+    category=EvaluationCategory.EXT_REQUEST_ADVISOR,
+    dataset_file="extensibility_request_advisor.jsonl",
+    entry_type=ExtRequestAdvisorEntry,
+    evaluators=("lm_checklist",),
+    core_score="test_passed",
+    runner="ubuntu-latest",
+    requires_container=False,
+    pass_bc_credentials=True,
+)

src/bcbench/categories/ext_request_implement/definition.py

@@ -0,0 +1,14 @@
+from bcbench.categories.definition import CategoryDefinition
+from bcbench.dataset import ExtRequestImplementEntry
+from bcbench.types import EvaluationCategory
+
+DEFINITION = CategoryDefinition(
+    category=EvaluationCategory.EXT_REQUEST_IMPLEMENT,
+    dataset_file="extensibility_request_implement.jsonl",
+    entry_type=ExtRequestImplementEntry,
+    evaluators=("lm_checklist",),
+    core_score="test_passed",
+    runner="ubuntu-latest",
+    requires_container=False,
+    pass_bc_credentials=True,
+)

src/bcbench/categories/ext_request_triage/definition.py

@@ -0,0 +1,14 @@
+from bcbench.categories.definition import CategoryDefinition
+from bcbench.dataset import ExtRequestTriageEntry
+from bcbench.types import EvaluationCategory
+
+DEFINITION = CategoryDefinition(
+    category=EvaluationCategory.EXT_REQUEST_TRIAGE,
+    dataset_file="extensibility_request_triage.jsonl",
+    entry_type=ExtRequestTriageEntry,
+    evaluators=("lm_checklist",),
+    core_score="test_passed",
+    runner="ubuntu-latest",
+    requires_container=False,
+    pass_bc_credentials=True,
+)

src/bcbench/categories/nl2al/definition.py

@@ -0,0 +1,14 @@
+from bcbench.categories.definition import CategoryDefinition
+from bcbench.dataset import NL2ALEntry
+from bcbench.types import EvaluationCategory
+
+DEFINITION = CategoryDefinition(
+    category=EvaluationCategory.NL2AL,
+    dataset_file="nl2al.jsonl",
+    entry_type=NL2ALEntry,
+    evaluators=("lm_checklist",),
+    core_score="test_passed",
+    runner="windows-latest",
+    requires_container=False,
+    pass_bc_credentials=True,
+)

src/bcbench/categories/test_generation/definition.py

@@ -0,0 +1,14 @@
+from bcbench.categories.definition import CategoryDefinition
+from bcbench.dataset import TestGenEntry
+from bcbench.types import EvaluationCategory
+
+DEFINITION = CategoryDefinition(
+    category=EvaluationCategory.TEST_GENERATION,
+    dataset_file="bcbench.jsonl",
+    entry_type=TestGenEntry,
+    evaluators=("resolution_rate", "build_rate", "pre_patch_failed_rate", "post_patch_passed_rate"),
+    core_score="ResolutionRate",
+    runner="GitHub-BCBench",
+    requires_container=True,
+    pass_bc_credentials=True,
+)

src/bcbench/cli_options.py

@@ -6,7 +6,8 @@
 import typer
 from bcbench_core.container import ContainerConfig
 
-from bcbench.types import AgentRuntimeConfig, EvaluationCategory
+from bcbench.categories import category_definition
+from bcbench.types import AgentRuntimeConfig, CopilotModelName, EvaluationCategory
 
 # Type aliases for cleaner command signatures
 # Note: Defaults are provided in function signatures, not here
@@ -109,22 +110,11 @@ def resolve_evaluation_runtime(
         al_lsp=al_lsp,
         bc_mcp=bc_mcp,
     )
-    if category.requires_container and runtime is None:
+    if category_definition(category).requires_container and runtime is None:
         raise typer.BadParameter(f"The {category.value} category requires a container", param_hint="--container-name")
     return runtime
 
 
-CopilotModelName = Literal[
-    "claude-sonnet-5",
-    "claude-opus-5",
-    "gpt-5.6-sol",
-    "gpt-5.6-terra",
-    "gpt-5.6-luna",
-    "gpt-5.3-codex",
-    "mai-code-1.1-flash",
-    "gemini-3.7-flash",
-]
-
 CopilotModel = Annotated[CopilotModelName, typer.Option(help="Copilot model to use")]
 
 ClaudeCodeModel = Annotated[

src/bcbench/commands/category.py

@@ -2,6 +2,7 @@
 
 import typer
 
+from bcbench.categories import category_definition
 from bcbench.cli_options import EvaluationCategoryOption
 from bcbench.github_actions import write_step_outputs
 from bcbench.types import EvaluationCategory
@@ -23,22 +24,24 @@ def bceval_config(category: EvaluationCategoryOption) -> None:
 
     The lines are appended to $GITHUB_OUTPUT so they become GitHub Actions step outputs. Outside of Actions nothing is written.
     """
+    definition = category_definition(category)
     outputs: dict[str, str] = {
-        "evaluators": ",".join(category.evaluators),
-        "core_score": category.core_score,
+        "evaluators": ",".join(definition.evaluators),
+        "core_score": definition.core_score,
     }
-    if ("lm_checklist" in category.evaluators) and (category.judge_model is not None):
+    if ("lm_checklist" in definition.evaluators) and (category.judge_model is not None):
         outputs["judge_model"] = category.judge_model
     write_step_outputs(outputs)
 
 
 @category_app.command("runtime-config")
 def runtime_config(category: EvaluationCategoryOption) -> None:
     """Emit the GitHub Actions runner label and environment requirements for a category."""
+    definition = category_definition(category)
     write_step_outputs(
         {
-            "runner": category.runner,
-            "requires-container": str(category.requires_container).lower(),
-            "requires-repo": str(category.requires_repo).lower(),
+            "runner": definition.runner,
+            "requires-container": str(definition.requires_container).lower(),
+            "requires-repo": str(definition.requires_repo).lower(),
         }
     )