feat(quickstart): narrate steps, per-stage timing, and always-on evidence - #616
Conversation
…ence Mirror the iii-hq/quickstart-validator output contract in the harness quickstart validator so every CI run shows what actually happened: - timestamped step narrative with [ok] assertions; installer and `iii worker add harness console` output streams live via tee while still landing in logs/ - per-stage durations in timings.tsv, a [timing breakdown] at the end (also on failure), and a `timings` array in result.json - an EVIDENCE.md digest (status, CLI version, timing, log tails, console::status, config.yaml, iii.lock) written by the cleanup trap on every outcome - workflow: EVIDENCE.md becomes the step summary, logs print in collapsible groups on every run, and the logs artifact uploads on success too (renamed harness-quickstart-logs)
ytallo
added
the
no-ticket
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
skill-check — worker0 verified, 49 skipped (no docs/).
Four for four. Nicely done. |
📝 WalkthroughWalkthroughThe quickstart validator now runs in timed stages, emits structured logs and assertions, records evidence on both success and failure, validates console and generated files, and exposes the resulting summaries and artifacts through GitHub Actions. ChangesQuickstart observability and staged validation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant QuickstartValidator
participant CLI
participant Engine
participant Console
QuickstartValidator->>CLI: Install CLI
QuickstartValidator->>Engine: Start empty engine
QuickstartValidator->>CLI: Add harness and console workers
QuickstartValidator->>Engine: Poll readiness and functions
QuickstartValidator->>Console: Check status and fetch root page
QuickstartValidator->>QuickstartValidator: Verify generated files
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (5)
.github/workflows/harness-quickstart.yml (2)
28-47: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueStep name understates what it does — it now also emits the evidence digest.
Consider renaming to
Show quickstart evidence and logsso the collapsed group is discoverable in the job log.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/harness-quickstart.yml around lines 28 - 47, Rename the workflow step currently named “Show quickstart logs” to “Show quickstart evidence and logs” so its label accurately reflects the EVIDENCE.md and log output it emits.
52-53: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
EVIDENCE.mdmay exceed the step-summary size limit.
GITHUB_STEP_SUMMARYis capped (1 MiB); the digest embeds a tail of every log plusconfig.yaml,iii.lock, andconsole-status.json, and the cap is configurable upward viaHARNESS_QUICKSTART_EVIDENCE_TAIL_LINES. When the cap is exceeded the summary is dropped for the whole step, losing the fallback table too. Consider truncating (e.g.head -c 900000) before appending.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/harness-quickstart.yml around lines 52 - 53, Limit the EVIDENCE.md content appended in the workflow step around the target/harness-quickstart/EVIDENCE.md check to stay below GitHub’s step-summary size limit, truncating it to a safe maximum before writing to GITHUB_STEP_SUMMARY. Preserve the existing fallback output and ensure the truncation remains effective when HARNESS_QUICKSTART_EVIDENCE_TAIL_LINES increases the evidence size.harness/tests/quickstart/README.md (1)
27-33: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDocument
HARNESS_QUICKSTART_EVIDENCE_TAIL_LINES.The new knob (default 80) controls how much of each log lands in
EVIDENCE.md; mentioning it here makes it discoverable alongside the rest of the output contract.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@harness/tests/quickstart/README.md` around lines 27 - 33, Update the quickstart output-contract documentation to mention the HARNESS_QUICKSTART_EVIDENCE_TAIL_LINES configuration knob, including its default of 80 and that it controls how many trailing lines from each log are included in EVIDENCE.md.harness/tests/quickstart/run-ci.sh (2)
155-158: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDead fallback branch now that
jqis a preflight requirement.Line 284 aborts the run when
jqis missing, so this branch is unreachable — and it would emit malformed JSON iffailure_reasoncontained a quote or newline. Consider dropping it (or keeping it and escaping viajq-free quoting) to avoid a misleading second serialization path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@harness/tests/quickstart/run-ci.sh` around lines 155 - 158, Remove the unreachable fallback serialization branch in the result-writing logic, since the jq preflight already guarantees jq is available. Keep the existing jq-based result generation path and its output behavior unchanged.
176-176: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value
&&one-liner isset -e-fragile despite the "never fail" contract.This currently only works because
cleanupdoesset +ebefore callingcollect_evidence. If the function is ever called from the main flow, an emptyrun_urlmakes the AND-list return 1 and aborts mid-digest. Prefer an explicitif.♻️ Defensive tweak
- [[ -n "$run_url" ]] && printf -- '- **CI run:** <%s>\n' "$run_url" + if [[ -n "$run_url" ]]; then + printf -- '- **CI run:** <%s>\n' "$run_url" + fi🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@harness/tests/quickstart/run-ci.sh` at line 176, Replace the && one-liner in collect_evidence with an explicit conditional that prints the CI run link only when run_url is non-empty, while ensuring the empty-value branch returns success under set -e. Preserve the existing output format and never-fail behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@harness/tests/quickstart/run-ci.sh`:
- Around line 419-421: Update the Console root fetch in the curl command to
include --retry-connrefused alongside the existing retry options, matching the
installer curl invocation, so connection-refused responses consume the
configured retry budget.
---
Nitpick comments:
In @.github/workflows/harness-quickstart.yml:
- Around line 28-47: Rename the workflow step currently named “Show quickstart
logs” to “Show quickstart evidence and logs” so its label accurately reflects
the EVIDENCE.md and log output it emits.
- Around line 52-53: Limit the EVIDENCE.md content appended in the workflow step
around the target/harness-quickstart/EVIDENCE.md check to stay below GitHub’s
step-summary size limit, truncating it to a safe maximum before writing to
GITHUB_STEP_SUMMARY. Preserve the existing fallback output and ensure the
truncation remains effective when HARNESS_QUICKSTART_EVIDENCE_TAIL_LINES
increases the evidence size.
In `@harness/tests/quickstart/README.md`:
- Around line 27-33: Update the quickstart output-contract documentation to
mention the HARNESS_QUICKSTART_EVIDENCE_TAIL_LINES configuration knob, including
its default of 80 and that it controls how many trailing lines from each log are
included in EVIDENCE.md.
In `@harness/tests/quickstart/run-ci.sh`:
- Around line 155-158: Remove the unreachable fallback serialization branch in
the result-writing logic, since the jq preflight already guarantees jq is
available. Keep the existing jq-based result generation path and its output
behavior unchanged.
- Line 176: Replace the && one-liner in collect_evidence with an explicit
conditional that prints the CI run link only when run_url is non-empty, while
ensuring the empty-value branch returns success under set -e. Preserve the
existing output format and never-fail behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 46048c86-686d-425c-8975-5c43c5a4beb2
📒 Files selected for processing (3)
.github/workflows/harness-quickstart.ymlharness/tests/quickstart/README.mdharness/tests/quickstart/run-ci.sh
| curl -fsS --retry 10 --retry-delay 1 \ | ||
| "http://127.0.0.1:$console_port/" -o "$artifact_dir/console.html" | ||
|
|
||
| ok "fetched Console root ($(wc -c <"$artifact_dir/console.html" | tr -d ' ') bytes)" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Add --retry-connrefused when fetching the Console root.
--retry alone does not retry ECONNREFUSED, so if the Console HTTP listener isn't bound yet the fetch fails on the first attempt and the run dies — the retry budget here is effectively unused. Line 368 already uses this flag for the installer.
🐛 Proposed fix
-curl -fsS --retry 10 --retry-delay 1 \
+curl -fsS --retry 10 --retry-connrefused --retry-delay 1 \
"http://127.0.0.1:$console_port/" -o "$artifact_dir/console.html"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| curl -fsS --retry 10 --retry-delay 1 \ | |
| "http://127.0.0.1:$console_port/" -o "$artifact_dir/console.html" | |
| ok "fetched Console root ($(wc -c <"$artifact_dir/console.html" | tr -d ' ') bytes)" | |
| curl -fsS --retry 10 --retry-connrefused --retry-delay 1 \ | |
| "http://127.0.0.1:$console_port/" -o "$artifact_dir/console.html" | |
| ok "fetched Console root ($(wc -c <"$artifact_dir/console.html" | tr -d ' ') bytes)" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@harness/tests/quickstart/run-ci.sh` around lines 419 - 421, Update the
Console root fetch in the curl command to include --retry-connrefused alongside
the existing retry options, matching the installer curl invocation, so
connection-refused responses consume the configured retry budget.
1 participant
Summary
CI-only change to the harness quickstart validator, mirroring the output contract of iii-hq/quickstart-validator so every run — pass or fail — shows exactly what happened. The pipeline exists to guarantee
iii worker add harness consoleworks; its output is now the protagonist of the job log.harness/tests/quickstart/run-ci.sh[HH:MM:SS] Step 3: iii worker add harness console) with[ok]assertion lines; installer andworker addoutput streams live in the CI log viateewhile still landing inlogs/.install_cli,start_engine,worker_add_harness_console,wait_for_functions,console_check,verify_files): each logs[timing] name: Ns, appends totimings.tsv, feeds atimingsarray inresult.json, and the run ends with an aligned[timing breakdown]— also on failure (the cleanup trap closes an interrupted stage so its partial duration still shows).EVIDENCE.mddigest written on every outcome: status, CLI version, install URL, commit, run link, failure reason, timing breakdown, 80-line tails of every log,console::status,config.yaml,iii.lock..github/workflows/harness-quickstart.ymlEVIDENCE.md(fallback to the old table).EVIDENCE.mdplus each log in collapsible groups on every run.harness-quickstart-logs; the result artifact addsEVIDENCE.mdandtimings.tsv.Testing
bash -nand YAML parse pass.result.json,EVIDENCE.mdcomposition).gh workflow run harness-quickstart.yml --ref feat/quickstart-evidence-logs— a dispatch validates end to end.Label
no-ticket: CI/validator tooling only, no product code touched.Summary by CodeRabbit
New Features
Bug Fixes
Documentation