Skip to content

feat(engine): add layer-based scenario selection (evaluation.scenarios_per_layer)#7

Merged
regevguym merged 1 commit intomondaycom:mainfrom
nymeria-ai:feat/layer-based-scenario-selection
Mar 22, 2026
Merged

feat(engine): add layer-based scenario selection (evaluation.scenarios_per_layer)#7
regevguym merged 1 commit intomondaycom:mainfrom
nymeria-ai:feat/layer-based-scenario-selection

Conversation

@nymeria-ai
Copy link
Copy Markdown
Contributor

What

Adds evaluation.scenarios_per_layer config to suite YAML schema, allowing suite authors to control how many scenarios are randomly selected per evaluation layer.

Why

Previously, scenario selection was either "run all" or done externally by consumers (e.g. agentalent's blind shuffle + slice). This led to unbalanced evaluations — potentially 3 execution scenarios and 0 reasoning.

Suite authors should control the evaluation structure.

How

New YAML config

evaluation:
  scenarios_per_layer:
    execution: 1        # pick 1 random from execution scenarios
    reasoning: 1        # pick 1 random from reasoning scenarios
    # self-improvement: omitted → all included

Rules

  • No evaluation config → ALL scenarios run (backwards compatible)
  • Layer omitted from scenarios_per_layer → all scenarios for that layer included
  • Layer with count → that many randomly picked (seeded PRNG for determinism)
  • Order follows YAML definition order (not hardcoded)

Changes

  • types.ts: New EvaluationConfig interface on SuiteDefinition
  • schema.ts: Zod validation for evaluation.scenarios_per_layer
  • loader.ts: New pickScenariosByLayer() function (exported)
  • index.ts: Export pickScenariosByLayer + EvaluationConfigSchema
  • ARCHITECTURE.md: Docs section + YAML example
  • 8 new tests covering all selection behaviors

Test results

Test Files  16 passed (16)
     Tests  226 passed (226)

Breaking changes

None. The evaluation field is optional with full backwards compatibility.

…s_per_layer)

- New SuiteDefinition.evaluation config with scenarios_per_layer
- pickScenariosByLayer() function: respects per-layer counts, preserves YAML order
- If no config → all scenarios run (backwards compatible)
- If layer omitted → all scenarios for that layer included
- Seeded PRNG for deterministic selection
- Zod schema validation for evaluation config
- 8 new tests covering all selection behaviors
- Updated ARCHITECTURE.md with docs and examples
@regevguym regevguym merged commit f78ec0b into mondaycom:main Mar 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants