Skip to content

Add an end-to-end Verify smoke matrix (5 agents x levels, cache-hit) from docs/TODO #28

Description

@AdvancedUno

Summary

The "Verify (end-to-end)" checklist in docs/TODO.md is still entirely unchecked. Turn it into automated smoke/regression tests so the full pipeline can't silently break.

Why it matters

The unit suite mocks the provider per-module, but nothing exercises the whole pipeline across all agents/levels as a regression guard. These checks were planned and never wired.

Current behavior

docs/TODO.md:67-70, all unchecked:

  • smoke matrix: 5 agents × {intern, senior} on a bundled resume + a real GitHub user
  • level calibration shifts expectations (intern vs senior; phd-applicant vs postdoc)
  • prompt-cache hit on the 2nd run (usage.cache_read_input_tokens > 0)

Proposed change

  • Add a mocked-provider smoke test that runs all five agents end-to-end and asserts a well-formed Report, plus that changing --level shifts the level-expectation text in the prompt.
  • Add a live, opt-in (env-gated) prompt-cache-hit assertion.

Acceptance criteria

  • all-agents end-to-end smoke test (mocked provider) running in CI
  • a level-calibration assertion
  • a documented opt-in cache-hit check

Identified during a code review of the repo. Complements #11 (golden-fixture calibration harness): this issue is pipeline smoke/regression, #11 is scoring calibration.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions