Summary
The "Verify (end-to-end)" checklist in docs/TODO.md is still entirely unchecked. Turn it into automated smoke/regression tests so the full pipeline can't silently break.
Why it matters
The unit suite mocks the provider per-module, but nothing exercises the whole pipeline across all agents/levels as a regression guard. These checks were planned and never wired.
Current behavior
docs/TODO.md:67-70, all unchecked:
- smoke matrix: 5 agents × {intern, senior} on a bundled resume + a real GitHub user
- level calibration shifts expectations (intern vs senior; phd-applicant vs postdoc)
- prompt-cache hit on the 2nd run (
usage.cache_read_input_tokens > 0)
Proposed change
- Add a mocked-provider smoke test that runs all five agents end-to-end and asserts a well-formed
Report, plus that changing --level shifts the level-expectation text in the prompt.
- Add a live, opt-in (env-gated) prompt-cache-hit assertion.
Acceptance criteria
Identified during a code review of the repo. Complements #11 (golden-fixture calibration harness): this issue is pipeline smoke/regression, #11 is scoring calibration.
Summary
The "Verify (end-to-end)" checklist in
docs/TODO.mdis still entirely unchecked. Turn it into automated smoke/regression tests so the full pipeline can't silently break.Why it matters
The unit suite mocks the provider per-module, but nothing exercises the whole pipeline across all agents/levels as a regression guard. These checks were planned and never wired.
Current behavior
docs/TODO.md:67-70, all unchecked:usage.cache_read_input_tokens > 0)Proposed change
Report, plus that changing--levelshifts the level-expectation text in the prompt.Acceptance criteria
Identified during a code review of the repo. Complements #11 (golden-fixture calibration harness): this issue is pipeline smoke/regression, #11 is scoring calibration.