(MOT-4269) fix(harness): harden reactive E2E validation

[ytallo]

Summary

• recognize actual overlap between writer sessions instead of requiring all spawn calls in one model response
• make the existing reactive scenario validations explicit around writer completion, aggregate upserts, report verification, root wake-up, and trigger cleanup

Root cause

The writers executed concurrently, but the evaluator only accepted three spawn calls emitted in the same response. The finalization sequence also allowed transient aggregate state, completion before writer status updates, and a wake-up that could create a child session instead of resuming the root.

Impact

The scenario now measures real concurrent writer activity and validates completion and cleanup deterministically while preserving the intended workload.

Validation

• cargo fmt --manifest-path harness/Cargo.toml --all -- --check
• cargo test --locked --manifest-path harness/Cargo.toml -p harness-e2e (47 passed)
• cargo test --locked --manifest-path harness/Cargo.toml -p harness (300 passed across unit, manifest, and schema tests)
• cargo clippy --locked --manifest-path harness/Cargo.toml -p harness -p harness-e2e -p harness-integration --all-targets -- -D warnings
• make -C harness integration-validate (11 fixtures valid)
• real-stack reactive_automation with zai/glm-5.2 (2/2 passed, score 100 each)
• git diff --check

Refs MOT-4269

Summary by CodeRabbit

• Improvements
  • Improved reactive automation validation for workflows with multiple concurrent writer sessions.
  • Added tracking for overlapping session activity to better verify parallel execution.
  • Strengthened aggregation checks to prevent double counting and transient incomplete results.
  • Enhanced finalization validation, including completion ordering, report accuracy, timing, and cleanup checks.
• Tests
  • Expanded end-to-end coverage for concurrent sessions, aggregation behavior, and final report correctness.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
workers	Ready	Preview, Comment	Jul 29, 2026 11:38am
workers-tech-spec	Ready	Preview, Comment	Jul 29, 2026 11:38am

[github-actions]

skill-check — worker

0 verified, 49 skipped (no docs/).

Layer	Result
structure	✓
vale	✓
ai	✓
render	✓

Four for four. Nicely done.

[coderabbitai]

Warning

Review limit reached

@ytallo, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 58 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ee407614-f98e-4ed1-bda7-954277fba36e

📥 Commits

Reviewing files that changed from the base of the PR and between 0d3d287 and 58b5c89.

📒 Files selected for processing (4)

• harness/tests/e2e/src/scenarios/reactive_automation/evaluate.rs
• harness/tests/e2e/src/scenarios/reactive_automation/evidence.rs
• harness/tests/e2e/src/scenarios/reactive_automation/prompt.rs
• harness/tests/e2e/src/scenarios/reactive_automation/queries.rs

📝 Walkthrough

Walkthrough

The reactive automation scenario now measures overlapping writer sessions, accepts concurrent-session evidence during scoring, and strengthens prompt requirements for aggregation, completion gating, report validation, root-session wakeup, and cleanup.

Changes

Reactive automation validation

Layer / File(s)	Summary
Aggregation and finalization orchestration harness/tests/e2e/src/scenarios/reactive_automation/prompt.rs	The prompt requires per-writer upserts, all-writer completion gating, validated report correction, root-session wakeup, and cleanup checks.
Concurrent session evidence harness/tests/e2e/src/scenarios/reactive_automation/queries.rs	Transcript timestamps are converted into activity windows, peak overlap is calculated, and overlapping/non-overlapping cases are tested.
Writer concurrency scoring contract harness/tests/e2e/src/scenarios/reactive_automation/evidence.rs, harness/tests/e2e/src/scenarios/reactive_automation/evaluate.rs	Writer evidence stores concurrent sessions, and scoring accepts either parallel calls or concurrent sessions matching the expected count.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant WriterSessions
  participant CompletionReaction
  participant Finalizer
  participant ReportRow
  participant RootSession
  WriterSessions->>CompletionReaction: mark all three writers done
  CompletionReaction->>Finalizer: invoke finalization
  Finalizer->>ReportRow: recompute totals and write validated report
  Finalizer->>RootSession: wake with root session id
  RootSession->>CompletionReaction: unregister run-scoped reactions

Loading

Poem

I’m a rabbit counting writers in flight,
Three little sessions overlap just right.
Totals hop upward, no doubles in sight,
The report gets checked before twilight.
Root wakes clean—what a delightful night!

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/harness-reactive-e2e-followup

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}