diff --git a/.gitignore b/.gitignore index 71e7ab2b..54a357d9 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,12 @@ _output # do not apply ignore rules to anything in vendor/ dir !vendor/ !vendor/** +# ─── OpenSpec / local tooling (install via openspec install.sh; do not commit) ─── +# Leading / = repo root only (do not ignore harness-evals/evals/). +/.cursor/ +/.work/ +/openspec/ +/eval-generation/ +/evals/ +/dashboard/ +# Operator-owned harness stays committed: harness-evals/{harness-docs,constitution.md,evals}/ diff --git a/AGENTS.md b/AGENTS.md index 10fa4662..565f2b23 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,6 +76,18 @@ harness-evals/harness-docs/ **AI Agent Path**: `harness-evals/harness-docs/*-guidelines.md` (as needed) → `domain/` → `architecture/` → `decisions/` → `ESO_DEVELOPMENT.md` +## OpenSpec (planning / evals) + +| Path | Role | +|------|------| +| `AGENTS.md` (this file) | Agent routing + architecture for `/opsx-*` | +| `harness-evals/harness-docs/` | Source docs for `/opsx-constitute` | +| `harness-evals/constitution.md` | Guardrails required before `plan.md` | +| `harness-evals/evals/` | Stage eval quality gates (`/opsx-continue`, `/opsx-apply`) | +| `eval-generation/` | Retrospective `/eval-loop` (fill `input/feature-bundle.yaml`) | + +See `harness-evals/README.md`. Evals are optional until `/eval-loop` populates cases. + ## Namespaces & Image Resolution | Namespace | Purpose | diff --git a/harness-evals/README.md b/harness-evals/README.md new file mode 100644 index 00000000..0b1fa808 --- /dev/null +++ b/harness-evals/README.md @@ -0,0 +1,25 @@ +# Harness evals (operator-owned) + +Operator-specific inputs for the [OpenSpec agile workflow](https://github.com/sujkini/openspec/blob/v2-restructured/README.md). + +``` +harness-evals/ +├── harness-docs/ # Operator docs (source for /opsx-constitute) +├── constitution.md # Guardrails (required before plan.md) +└── evals/ # Stage eval YAMLs (quality gates; optional until /eval-loop) +``` + +| Path | Command | Notes | +|------|---------|-------| +| `harness-docs/` | `/opsx-constitute` | Already populated with ESO guidelines | +| `constitution.md` | `/opsx-continue` (before plan) | Present — regenerate with `/opsx-constitute` if needed | +| `evals/*_eval.yaml` | `/opsx-continue`, `/opsx-apply` | Stub files seeded; cases accumulate via `/eval-loop` | + +## Populate real eval cases + +1. Fill `eval-generation/input/feature-bundle.yaml` from a **completed** feature (EP, epic, stories, PRs, bugs). +2. Run `/eval-loop`. +3. Review `eval-generation/eval-generation-workflow/template-gaps/` and `eval-generation/output-refined-templates/`. +4. Generated cases sync automatically into `harness-evals/evals/`. + +Until cases exist, forward workflow skips eval scoring and relies on verification + user approval. diff --git a/harness-evals/constitution.md b/harness-evals/constitution.md new file mode 100644 index 00000000..b948b2eb --- /dev/null +++ b/harness-evals/constitution.md @@ -0,0 +1,142 @@ + +# External Secrets Operator Constitution + +**AgentRoutingMode:** PROVIDED + + +**Version**: 1.0.0 | **Ratified**: 2026-07-01 | **Last Amended**: 2026-07-01 + +## Core Principles + +### I. Upstream Operand Separation — Do Not Fork Upstream Logic + +The operator deploys and manages upstream **external-secrets** and the optional **bitwarden-sdk-server** plugin via **embedded manifests in `bindata/external-secrets/`**. The operator **never** reimplements upstream secret-sync logic (provider authentication, ExternalSecret reconciliation, generator behavior, Bitwarden SDK protocol). Operator packages reconcile operator CRs and deploy/configure operand workloads only. + +**Evidence:** `bindata/external-secrets/resources/` — operand YAML from upstream helm; `pkg/controller/external_secrets/` installs deployments/RBAC/webhooks but contains zero provider-specific secret-fetch logic. `README.md` states the operator uses upstream helm charts. + +### II. Two Operand Workloads — Core vs Plugin + +| Workload | Always? | Controlled by | Image env | +|----------|---------|---------------|-----------| +| **external-secrets** (core controller + webhook + cert-controller) | Yes | `ExternalSecretsConfig` | `RELATED_IMAGE_EXTERNAL_SECRETS` | +| **bitwarden-sdk-server** (provider plugin) | No — when `plugins.bitwardenSecretManagerProvider.mode == Enabled` | `ExternalSecretsConfig.spec.plugins` | `RELATED_IMAGE_BITWARDEN_SDK_SERVER` | + +New plugin workloads MUST follow the bitwarden pattern: API under `spec.plugins`, bindata deployment asset, conditional entry in `createOrApplyDeployments`, dedicated network policy, TLS via `certProvider` or `secretRef`. + +**Evidence:** `deployments.go` — conditional deployment table; `constants.go` — `bitwardenDeploymentAssetName`, image env var names; `api/v1alpha1/external_secrets_config_types.go` — `BitwardenSecretManagerProvider`, CEL rules for TLS prerequisites. + +### III. Controller-Runtime Only — Single Manager, Three Reconcilers + +All controllers use **`sigs.k8s.io/controller-runtime`** on **one** shared manager. Register reconcilers in `pkg/operator/setup_manager.go` only. Do not introduce library-go, informer factories, or separate managers. + +**Evidence:** `pkg/operator/setup_manager.go` — wires `external_secrets_manager`, `external_secrets`, and optional `crd_annotator`; `cmd/external-secrets-operator/main.go` — single `ctrl.Manager`; `go.mod` — `sigs.k8s.io/controller-runtime v0.23.3`, no `openshift/library-go`. + +### IV. Create-or-Update Reconciliation — Not SSA-First + +Operand resources are reconciled via **create-or-update with deep equality** (`createWithFallback`, `common.HasObjectChanged`, `UpdateWithRetry`). Limited SSA (`client.Apply` with field owner) is allowed only for CR annotation patches. Do not convert operand reconcilers to SSA-first patterns. + +**Evidence:** `pkg/controller/external_secrets/install_external_secrets.go`, `pkg/controller/common/utils.go` — `HasObjectChanged()`; `pkg/controller/common/constants.go` — `ExternalSecretsOperatorCommonName` as field owner for annotation patches only. + +### V. Singleton CR Convention — Name `cluster`, One Per Kind + +Operator CRs `ExternalSecretsConfig` and `ExternalSecretsManager` are **cluster-scoped singletons named `cluster`**. The operator auto-creates default `ExternalSecretsManager` named `cluster`. CEL validation enforces singleton naming. + +**Evidence:** `pkg/controller/common/constants.go` — `ExternalSecretsConfigObjectName` and `ExternalSecretsManagerObjectName` = `"cluster"`; `pkg/operator/setup_manager.go` — `CreateDefaultESMResource()`; `README.md` — auto-creates `externalsecretsmanagers.operator.openshift.io` named `cluster`. + +### VI. Feature Flags on ExternalSecretsManager — Not OpenShift FeatureGate API + +Runtime feature toggles are defined on `ExternalSecretsManager.Spec.Features[]` with typed `FeatureName` values. Check via `common.IsFeatureEnabled()`. Do not add OpenShift cluster FeatureGate discovery or `pkg/features/` patterns from other operators. + +**Evidence:** `api/v1alpha1/external_secrets_manager_types.go` — `Feature` slice; `pkg/controller/common/utils.go` — `IsFeatureEnabled()`; `pkg/controller/external_secrets/constants.go` — `featureContainerArgs` map. + +### VII. Webhook TLS — cert-manager or In-Tree cert-controller (Mutually Exclusive) + +Webhook TLS uses either cert-manager `Certificate` CRs (`certProvider.certManager.mode == Enabled`) **or** the in-tree `external-secrets-cert-controller` deployment — never both. cert-controller deployment is skipped when cert-manager path is active. + +**Evidence:** `deployments.go` — `certControllerDeploymentAssetName` condition `!isCertManagerConfigEnabled(esc)`; `certificate.go`, `certificate_external-secrets-webhook.yml` vs `secret_external-secrets-webhook.yml`. + +### VIII. Bindata / Manifest Regeneration — Never Hand-Edit, Always `make update` + +Operand manifests under `bindata/` and generated code (`zz_generated.deepcopy.go`, `pkg/operator/assets/bindata.go`, CRD YAML under `config/crd/bases/`) are **generated artifacts**. Changes require `make update-operand-manifests` (helm pipeline) and/or `make generate && make manifests && make update-bindata`. CI verification (`make verify`) fails if outputs are stale. + +**Evidence:** `hack/update-external-secrets-manifests.sh`; `Makefile` — `EXTERNAL_SECRETS_VERSION`, `update`, `verify`, `verify-bindata`, `verify-generated`; `pkg/operator/assets/bindata.go` — generated. + +### IX. Verification-First Development — `make verify && make lint && make test` + +All changes MUST pass: `make test` (manifests, generate, fmt, vet, test-apis, test-unit), `make verify` (bindata, generated, govulncheck, git diff), and `make lint` (golangci-lint + kube-api-linter). E2E (`make test-e2e`, build tag `e2e`) requires a live cluster. + +**Evidence:** `Makefile` — `test`, `test-unit`, `test-apis`, `test-e2e`, `verify`, `lint` targets; `.golangci.yml` — linter configuration with kube-api-linter plugin. + +### X. RBAC Least Privilege — Explicit Operator and Operand Manifests + +Operator RBAC is in `config/rbac/`. Operand RBAC is embedded in `bindata/external-secrets/resources/` and applied by `pkg/controller/external_secrets/rbacs.go`. New permissions MUST be explicit ClusterRole rules in bindata or operator RBAC — not broad cluster-admin grants. + +**Evidence:** `config/rbac/role.yaml`; `bindata/external-secrets/resources/` — per-component RBAC YAML; `pkg/controller/external_secrets/rbacs.go`. + +### XI. OLM Bundle and Related Images + +The operator ships via OLM (`bundle/`, `config/manifests/`). Operand version is pinned in `Makefile` (`EXTERNAL_SECRETS_VERSION`). Images: +- `RELATED_IMAGE_EXTERNAL_SECRETS` / `OPERAND_EXTERNAL_SECRETS_IMAGE_VERSION` — core operand +- `RELATED_IMAGE_BITWARDEN_SDK_SERVER` / `BITWARDEN_SDK_SERVER_IMAGE_VERSION` — Bitwarden plugin + +Missing image env vars cause irrecoverable errors. + +**Evidence:** `Makefile` — `IMG_VERSION`, `EXTERNAL_SECRETS_VERSION`, `bundle` target; `bundle/manifests/openshift-external-secrets-operator.clusterserviceversion.yaml`; `pkg/controller/external_secrets/constants.go`. + +### XII. Namespace Conventions + +Operator runs in `external-secrets-operator` namespace. Operand runs in `external-secrets` namespace (`OperandDefaultNamespace`). Both are established conventions in controller constants and README. + +**Evidence:** `README.md` — "operator runs in `external-secrets-operator` namespace"; `pkg/controller/external_secrets/constants.go` — `OperandDefaultNamespace`. + +## Additional Constraints + +- **Go version**: Match `go.mod` — currently `go 1.26.0`. — **Evidence:** `go.mod` +- **Workspace**: Multi-module `go.work` (root, `cmd/external-secrets-operator`, `test`, `tools`). Vendor via `make update-vendor`. — **Evidence:** `go.work`, `vendor/` +- **Import ordering**: Local prefix `github.com/openshift/external-secrets-operator`. — **Evidence:** `.golangci.yml` `local-prefixes` +- **FIPS**: Production builds use `hack/go-fips.sh` (`GOEXPERIMENT=strictfipsruntime`, `CGO_ENABLED=1`). — **Evidence:** `Makefile` `build-operator`, `hack/go-fips.sh` +- **Container image**: Operator image from `Dockerfile` / `images/ci/`; operand images from `RELATED_IMAGE_*` env vars. — **Evidence:** `Dockerfile`, CSV relatedImages +- **Test frameworks**: Standard `testing` + counterfeiter fakes in `pkg/`; Ginkgo v2 + envtest in `test/apis/`; Ginkgo + live cluster in `test/e2e/` (tag `e2e`). — **Evidence:** `pkg/controller/client/fakes/`, `test/apis/`, `test/e2e/` +- **CI system**: Prow via `openshift/release`; in-repo verify via `make verify`. — **Evidence:** `README.md` contributing section +- **Optional cert-manager**: Webhook TLS may use cert-manager `Certificate` CRs when cert-manager is installed; `crd_annotator` is conditional. Never assume cert-manager is present. — **Evidence:** `pkg/operator/setup_manager.go`, `pkg/controller/crd_annotator/` + +## Development Workflow + +| Activity | Requirement | Evidence | +|----------|-------------|----------| +| Local unit tests | `make test-unit` or full `make test` | `Makefile` | +| API validation tests | `make test-apis` after CRD/testsuite changes | `hack/test-apis.sh`, `test/apis/` | +| Full verify | `make verify` | `Makefile` `verify` target | +| Lint | `make lint` | `Makefile`, `.golangci.yml` | +| Codegen refresh | `make generate && make manifests` after API edits | `Makefile` | +| Operand bump | `make update-operand-manifests && make update-bindata` | `hack/update-external-secrets-manifests.sh` | +| E2E tests | `make test-e2e` (cluster required); filter via `E2E_GINKGO_LABEL_FILTER` | `Makefile` `test-e2e` | +| Bundle generation | `make bundle` after CSV/CRD changes | `Makefile` `bundle` | +| PR pre-merge | `make test && make verify && make lint`; commit generated outputs | `AGENTS.md` verification matrix | + +## Agent Routing + +| Agent ID | Scope | When to route | +|----------|-------|---------------| +| **API_Agent** | `api/v1alpha1/`, testsuite YAML | CRD types, validation, markers | +| **OperatorController_Agent** | `pkg/controller/external_secrets/`, `external_secrets_manager/`, `crd_annotator/`, `setup_manager.go` | Core operand reconciliation, wiring | +| **ManifestsBindata_Agent** | `bindata/`, `hack/update-external-secrets-manifests.sh`, operand CRDs | Operand manifest refresh, version pins | +| **BitwardenPlugin_Agent** | `Plugins.BitwardenSecretManagerProvider`, bitwarden bindata assets | Bitwarden SDK plugin workload | +| **WebhookTLS_Agent** | `certificates.go`, webhook deployments, trusted CA | Webhook TLS paths | +| **RBACSecurity_Agent** | `config/rbac/`, `rbacs.go`, `networkpolicy.go` | RBAC and network policy | +| **OLMRelease_Agent** | `bundle/`, `config/manifests/` | CSV, relatedImages | +| **Testing_Agent** | `test/e2e/`, `test/apis/` | Test authoring | +| **Docs_Agent** | `README.md`, `docs/` | User-facing docs | + +Full routing detail: `AGENTS.md` (repo root). + +## Governance + +- This constitution supersedes ad-hoc conventions for downstream Planning, Task Creation, and Code Generation agents. +- **Amendments:** require documented evidence of repo change; bump Version and Last Amended date. +- **Conflicts:** if spec contradicts constitution, escalate in plan.md §8 — do not silently override. Forking upstream external-secrets logic into the operator is a constitution violation. +- **Companion docs:** + - **AGENTS.md** takes precedence for agent routing, controller map, Make targets, and test patterns. + - **README.md** takes precedence for human-facing install and contributing procedures. + - **This constitution** takes precedence for architectural principles and non-negotiable guardrails. +- **Complexity:** new patterns must justify deviation from existing repo conventions. Adding a second controller framework or SSA-first operand reconciliation requires constitution amendment. diff --git a/harness-evals/evals/README.md b/harness-evals/evals/README.md new file mode 100644 index 00000000..9217ef91 --- /dev/null +++ b/harness-evals/evals/README.md @@ -0,0 +1,16 @@ +# Stage evals (quality gates) + +Operator-owned eval rubrics used by `/opsx-continue` and `/opsx-apply`. + +| File | Used by | +|------|---------| +| `repo-assessment_eval.yaml` | `/opsx-continue` (repo-assessment) | +| `plan_eval.yaml` | `/opsx-continue` (plan) | +| `tasks_eval.yaml` | `/opsx-continue` (tasks) | +| `code-generation_eval.yaml` | `/opsx-apply` (per-task, ai-helpers mode) | + +- Empty `evals: []` means the stage gate is present but has no cases yet — scoring is a no-op until `/eval-loop` populates cases. +- `/eval-loop` auto-syncs generated cases from `eval-generation/output-evals/` into this directory. +- Do not edit schema package evals for forward workflow; this directory is the source of truth. + +See: https://github.com/sujkini/openspec/blob/v2-restructured/README.md diff --git a/harness-evals/evals/code-generation_eval.yaml b/harness-evals/evals/code-generation_eval.yaml new file mode 100644 index 00000000..6c88756e --- /dev/null +++ b/harness-evals/evals/code-generation_eval.yaml @@ -0,0 +1,152 @@ +stage: code-generation +artifact: fork working copy (code + tests) +version: 1 +eval_count: 8 +oape_commands: + - api-generate + - api-generate-tests + - api-implement + - e2e-generate + - manual + - any +evals: + - id: cg-r1-001-api-reference-policy-enum + round: 1 + stage: code-generation + oape_command: api-generate + source_issue_ids: [ESO-435, PR-2011-review-optional] + prompt: > + New optional ConfigMap/Secret reference types use an explicit Required/Optional policy enum + (or OpenShift-equivalent) rather than nested boolean optionality; defaults are documented. + assertions: + - type: must_mention + value: reference policy enum or equivalent non-boolean optionality + - type: must_mention + value: CRD/defaulting for key or policy fields when applicable + scoring: + weight: 1.0 + pass_threshold: 0.75 + + - id: cg-r1-002-api-reserve-operator-env + round: 1 + stage: code-generation + oape_command: api-generate + source_issue_ids: [ESO-435, ESO-417] + prompt: > + When the operator owns env var names for injected trust/config, the CRD CEL (or equivalent) + rejects those names on user overrideEnv-style fields in the same API change. + assertions: + - type: must_mention + value: CEL or validation rule reserving operator-owned env names + scoring: + weight: 1.0 + pass_threshold: 0.8 + + - id: cg-r1-003-api-tests-cel-and-defaults + round: 1 + stage: code-generation + oape_command: api-generate-tests + source_issue_ids: [ESO-435] + prompt: > + API integration tests cover defaults for reference key/policy and rejection of reserved + operator-owned env names via override-style fields. + assertions: + - type: must_mention + value: test cases for defaults on reference fields + - type: must_mention + value: test cases for CEL rejection of reserved env names + scoring: + weight: 1.0 + pass_threshold: 0.75 + + - id: cg-r1-004-validate-before-deploy-patch + round: 1 + stage: code-generation + oape_command: api-implement + source_issue_ids: [ESO-396, PR-149-review-validation] + prompt: > + User-supplied trust/config material is validated before Deployment mutation; invalid material + maps to typed user-configuration errors and Degraded status without applying a broken trust patch. + assertions: + - type: must_mention + value: validation precedes Deployment update/patch + - type: must_mention + value: distinct error or event reasons for invalid material classes + - type: must_mention + value: Degraded or UserConfiguration-style status mapping + scoring: + weight: 1.0 + pass_threshold: 0.8 + + - id: cg-r1-005-watch-label-self-heal + round: 1 + stage: code-generation + oape_command: api-implement + source_issue_ids: [ESO-396, ESO-237] + prompt: > + Referenced ConfigMaps/Secrets are watched (watching label or equivalent) so data fixes + requeue reconcile; operator-created co-managed ConfigMaps use update/reconcile paths that + tolerate ownership/label drift without permanent AlreadyExists dead-ends. + assertions: + - type: must_mention + value: watch or label for referenced user ConfigMap/Secret + - type: must_mention + value: update or reconcile path for co-managed ConfigMaps when applicable + scoring: + weight: 1.0 + pass_threshold: 0.75 + + - id: cg-r1-006-component-scoped-mount + round: 1 + stage: code-generation + oape_command: api-implement + source_issue_ids: [ESO-396] + prompt: > + Volume/env mutations for controller configuration apply only to in-scope operand components; + non-target Deployments remain unchanged. + assertions: + - type: must_mention + value: mutation gated to specific component or Deployment + - type: must_mention + value: unit tests asserting non-target components lack the volume or env + scoring: + weight: 0.9 + pass_threshold: 0.75 + + - id: cg-r1-007-interaction-skip-path + round: 1 + stage: code-generation + oape_command: api-implement + source_issue_ids: [ESO-396, PR-149-review-validation] + prompt: > + When an alternate injection path is already active (e.g. proxy/CNO bundle), the controller + skips duplicate mounts and emits an observable event or log reason while remaining Ready. + assertions: + - type: must_mention + value: skip path when alternate injection already satisfies trust + - type: must_mention + value: event or reason documenting the skip + scoring: + weight: 0.9 + pass_threshold: 0.75 + + - id: cg-r1-008-e2e-recovery-and-negatives + round: 1 + stage: code-generation + oape_command: e2e-generate + source_issue_ids: [ESO-396] + prompt: > + E2E coverage includes happy path, invalid material Degraded cases, recover-on-ConfigMap-fix + without CR edit, delete-after-success Degraded, component exclusion, and alternate-path skip matrix. + assertions: + - type: must_mention + value: recovery without CR change after fixing referenced object + - type: must_mention + value: Degraded scenarios for missing or invalid referenced material + - type: must_mention + value: negative check for non-target components + - type: must_mention + value: coexistence or skip with alternate injection path + scoring: + weight: 1.0 + pass_threshold: 0.75 diff --git a/harness-evals/evals/plan_eval.yaml b/harness-evals/evals/plan_eval.yaml new file mode 100644 index 00000000..7020234f --- /dev/null +++ b/harness-evals/evals/plan_eval.yaml @@ -0,0 +1,103 @@ +stage: plan +template: templates/plan-template.md +version: 1 +eval_count: 5 +evals: + - id: plan-r1-001-error-status-matrix + round: 1 + stage: plan + source_issue_ids: [ESO-396, PR-149-review-validation] + input_refs: + - pattern: user_pki_material_validation_degraded + prompt: > + When the plan consumes user-referenced ConfigMaps/Secrets, does it include an explicit + error/status matrix covering NotFound requeue, invalid content Degraded/events, and + optional-missing policy before implementation phases? + assertions: + - type: must_mention + value: NotFound or missing reference requeue behavior + - type: must_mention + value: invalid content Degraded or equivalent user-configuration failure + - type: must_cover_phase + value: error or status matrix appears before or within early implementation phases + scoring: + weight: 1.0 + pass_threshold: 0.8 + + - id: plan-r1-002-watch-self-healing + round: 1 + stage: plan + source_issue_ids: [ESO-396, ESO-237] + input_refs: + - pattern: user_referenced_object_watch_and_recovery + - pattern: co_managed_configmap_ownership_label_drift + prompt: > + Does the plan specify watch/label self-healing so referenced object fixes recover Ready + without CR edits, and cleanup when references are removed after success? + assertions: + - type: must_mention + value: watch or reconcile on referenced object changes + - type: must_mention + value: recovery without CR edit or equivalent self-healing + - type: must_align_with_constitution + value: does not invent SSA; stays consistent with UpdateWithRetry / existing client patterns from assessment + scoring: + weight: 1.0 + pass_threshold: 0.75 + + - id: plan-r1-003-reserved-operator-knobs + round: 1 + stage: plan + source_issue_ids: [ESO-435, ESO-417] + input_refs: + - pattern: operator_owned_env_cel_reservation + prompt: > + If the operator injects env/volume trust semantics, does the plan require API-level + reservation (CEL or equivalent) of operator-owned names in the contracts/API phase? + assertions: + - type: must_mention + value: CEL or API validation reserving operator-owned env or mount knobs + - type: must_cover_phase + value: API or contracts phase includes reservation before or with injection work + scoring: + weight: 1.0 + pass_threshold: 0.75 + + - id: plan-r1-004-interaction-matrix + round: 1 + stage: plan + source_issue_ids: [ESO-417, PR-149-review-validation] + input_refs: + - pattern: interaction_matrix_with_existing_trust_paths + prompt: > + Relative to alternate mechanisms listed in repo-assessment, does the plan document + skip/coexist/merge behavior and rejected cluster-wide-only alternatives where applicable? + assertions: + - type: must_mention + value: interaction or coexistence with existing injection or trust mechanisms + - type: must_mention + value: skip duplicate or merge semantics when overlapping mechanisms apply + scoring: + weight: 0.9 + pass_threshold: 0.75 + + - id: plan-r1-005-component-scope-and-api-conventions + round: 1 + stage: plan + source_issue_ids: [ESO-435, PR-2011-review-optional] + input_refs: + - pattern: component_scoped_deployment_mutation + - pattern: api_convention_enums_over_booleans + prompt: > + Does the plan explicitly scope which operand components receive the mutation and prefer + reference-policy enums over nested boolean optionality for optional references? + assertions: + - type: must_mention + value: which operand components are in scope and which are excluded + - type: must_mention + value: Required/Optional policy enum or equivalent non-boolean reference policy when optionality is needed + - type: must_reference_spec_ids + value: ties component scope or API shape back to spec/enhancement requirements + scoring: + weight: 0.9 + pass_threshold: 0.75 diff --git a/harness-evals/evals/repo-assessment_eval.yaml b/harness-evals/evals/repo-assessment_eval.yaml new file mode 100644 index 00000000..0ab3ac26 --- /dev/null +++ b/harness-evals/evals/repo-assessment_eval.yaml @@ -0,0 +1,82 @@ +stage: repo-assessment +template: templates/repo-assessment-template.md +version: 1 +eval_count: 4 +evals: + - id: ra-r1-001-comanaged-config-inventory + round: 1 + stage: repo-assessment + source_issue_ids: [ESO-237, ESO-396] + input_refs: + - pattern: co_managed_configmap_ownership + - pattern: user_referenced_object_watch_and_recovery + prompt: > + When a feature creates or references ConfigMaps/Secrets in the operand namespace, + does repo-assessment inventory co-managed vs user-referenced objects, ownership/watching + labels, and create-vs-update/reconcile paths with repo evidence? + assertions: + - type: must_identify_pattern + value: co-managed or user-referenced ConfigMap/Secret inventory with ownership or watching labels + - type: must_cite_repo_evidence + value: cites existing controller cache/watch or ConfigMap helpers as evidence + - type: must_cover_section + value: managed resources or watch/cache inventory discusses update-vs-create risk + scoring: + weight: 1.0 + pass_threshold: 0.75 + + - id: ra-r1-002-alternate-trust-paths + round: 1 + stage: repo-assessment + source_issue_ids: [RFE-8685, GH-124, ESO-417] + input_refs: + - pattern: missing_cr_scoped_trust_path + - pattern: interaction_matrix_with_existing_trust_paths + prompt: > + For TLS-facing or secret-backend connectivity features, does repo-assessment document + existing trust/injection mechanisms (system trust, proxy/CNO-style injection, per-resource + CA options) and the gap that motivates any new CR-scoped configuration? + assertions: + - type: must_identify_pattern + value: existing alternate trust or CA injection mechanisms + - type: must_mention + value: gap or limitation of cluster-wide-only or per-store-only approaches when relevant + scoring: + weight: 1.0 + pass_threshold: 0.75 + + - id: ra-r1-003-component-applicability + round: 1 + stage: repo-assessment + source_issue_ids: [ESO-396] + input_refs: + - pattern: component_scoped_deployment_mutation + prompt: > + When proposing operand Deployment mutations via controller configuration, does assessment + list which operand components/containers are valid targets and which must remain unchanged? + assertions: + - type: must_identify_pattern + value: per-component applicability for controllerConfig-like mutations + - type: must_mention + value: non-target components that must not receive the mutation + scoring: + weight: 0.8 + pass_threshold: 0.75 + + - id: ra-r1-004-error-classification-hooks + round: 1 + stage: repo-assessment + source_issue_ids: [ESO-396] + input_refs: + - pattern: user_pki_material_validation_degraded + prompt: > + Does repo-assessment cite existing error-classification / status-condition patterns + (Irrecoverable, Retry, UserConfiguration/Degraded) that a user-config feature should reuse? + assertions: + - type: must_cite_repo_evidence + value: points to existing error types or status condition helpers + - type: must_identify_pattern + value: user configuration errors map to Degraded or requeue semantics + scoring: + weight: 0.8 + pass_threshold: 0.75 diff --git a/harness-evals/evals/tasks_eval.yaml b/harness-evals/evals/tasks_eval.yaml new file mode 100644 index 00000000..348c0a9b --- /dev/null +++ b/harness-evals/evals/tasks_eval.yaml @@ -0,0 +1,75 @@ +stage: tasks +template: templates/tasks-template.md +version: 1 +eval_count: 4 +evals: + - id: tasks-r1-001-api-before-controller + round: 1 + stage: tasks + source_issue_ids: [ESO-435, ESO-396] + input_refs: + - pattern: enhancement_vs_ticket_field_name_drift + prompt: > + When a feature adds CR fields and reconcile behavior, does the task DAG sequence API/CRD + generation and API tests before controller tasks that consume the new fields? + assertions: + - type: must_respect_dag + value: API or CRD tasks precede controller implementation tasks that read the new fields + - type: must_assign_agent + value: API_Agent or equivalent owns API tasks; controller agent owns reconcile tasks + scoring: + weight: 1.0 + pass_threshold: 0.8 + + - id: tasks-r1-002-recovery-acceptance + round: 1 + stage: tasks + source_issue_ids: [ESO-396] + input_refs: + - pattern: user_referenced_object_watch_and_recovery + prompt: > + Do controller task acceptance criteria include recover-on-referenced-object-fix without + CR change and Degraded when the reference disappears after success? + assertions: + - type: must_define_acceptance_criteria + value: recovery when referenced ConfigMap/Secret is fixed without CR edit + - type: must_define_acceptance_criteria + value: Degraded or equivalent when required reference is deleted after success + scoring: + weight: 1.0 + pass_threshold: 0.75 + + - id: tasks-r1-003-component-exclusion-acceptance + round: 1 + stage: tasks + source_issue_ids: [ESO-396] + input_refs: + - pattern: component_scoped_deployment_mutation + prompt: > + Do acceptance criteria assert that non-target operand components do not receive the new + volume or env mutation? + assertions: + - type: must_define_acceptance_criteria + value: negative assertion for non-target Deployments or containers + scoring: + weight: 0.9 + pass_threshold: 0.75 + + - id: tasks-r1-004-cel-api-tests + round: 1 + stage: tasks + source_issue_ids: [ESO-435] + input_refs: + - pattern: operator_owned_env_cel_reservation + - pattern: api_convention_enums_over_booleans + prompt: > + Do API tasks that reserve operator-owned env names or add reference-policy enums include + API integration test coverage in acceptance criteria? + assertions: + - type: must_define_acceptance_criteria + value: API tests cover CEL rejection of reserved env names or policy/defaults behavior + - type: must_cover_all_spec_requirements + value: API contract requirements from the plan/spec are mapped to at least one task + scoring: + weight: 1.0 + pass_threshold: 0.75