feat(quickstart): z.ai live message check through the Console + next channel - #617
Merged
Conversation
…channel Extend the quickstart validator beyond install/boot: when ZAI_API_KEY is set, it now proves a real model round-trip through the Console. - Step 7: `iii worker add provider-zai`, wait for provider::zai::stream - Step 8: console_send.py speaks the engine WS protocol (invokefunction/invocationresult) through the Console /ws proxy — the same path the browser uses — sending harness::send (glm-5.2 via zai), polling harness::status to completion, and asserting a non-empty assistant reply from session::messages - result.json gains channel + message_check; EVIDENCE.md gains the prompt/reply section and drops the iii.lock, discovery.log and console-status.log noise - III_CHANNEL=next installs from the next channel; exposed as the `channel` choice input on workflow_dispatch - without ZAI_API_KEY the live check is skipped and recorded as skipped
ytallo
added
the
no-ticket
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughThe PR adds channel-aware quickstart validation, optional Console-to-provider messaging checks, richer evidence artifacts, and updates reactive automation references from ChangesQuickstart validation
Reactive automation trigger rename
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Validator
participant Console
participant Engine
participant ZAI
Validator->>Console: Send prompt through WebSocket
Console->>Engine: Start assistant turn
Engine->>ZAI: Request streamed response
Validator->>Console: Poll status and fetch messages
Console-->>Validator: Return completed turn and assistant reply
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
skill-check — worker0 verified, 49 skipped (no docs/).
Four for four. Nicely done. |
… actually registers The database worker registers its trigger type as database::row-changed (ROW_CHANGED_TYPE in database/src/triggers/bus.rs), but the e2e stack script polled for database::row-change and timed out after 60s, failing every scenario job on main (run 30395783495). Align the wait, the reactive_automation prompt/queries, and the README to the real name.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
.github/workflows/harness-quickstart.yml (1)
27-27: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winConsider
persist-credentials: falseon checkout.Static analysis flags credential persistence: this job doesn't need the checked-out repo to retain git credentials afterward.
🔒 Suggested fix
- uses: actions/checkout@v5 + with: + persist-credentials: false🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/harness-quickstart.yml at line 27, Update the actions/checkout@v5 step in the workflow to disable persisted Git credentials by configuring persist-credentials as false.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@harness/tests/quickstart/run-ci.sh`:
- Around line 203-210: Update the console-message evidence block in run-ci.sh to
render only when console-send.json contains valid JSON with a usable reply,
rather than merely when the file exists. Use jq validation or an equivalent
guard before printing the heading and reply, while preserving the existing
output for successful console_send.py runs and avoiding suppressed jq errors
being presented as evidence.
- Around line 489-525: Replace the asynchronous process substitution on the
console_send.py stderr redirection in the console_message_check stage with a
synchronous logging approach that preserves stderr output in console-send.log
before the failure path calls die. Keep stderr visible to the terminal and
ensure the complete diagnostics are available for collect_evidence and artifact
upload.
---
Nitpick comments:
In @.github/workflows/harness-quickstart.yml:
- Line 27: Update the actions/checkout@v5 step in the workflow to disable
persisted Git credentials by configuring persist-credentials as false.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3b74d1a6-be95-417e-a5c2-4d3f7db8595b
📒 Files selected for processing (8)
.github/workflows/harness-quickstart.ymlharness/tests/e2e/README.mdharness/tests/e2e/run-ci.shharness/tests/e2e/src/scenarios/reactive_automation/prompt.rsharness/tests/e2e/src/scenarios/reactive_automation/queries.rsharness/tests/quickstart/README.mdharness/tests/quickstart/console_send.pyharness/tests/quickstart/run-ci.sh
Comment on lines
+203
to
+210
| if [[ -f "$artifact_dir/console-send.json" ]]; then | ||
| printf '## Console message check (%s via %s)\n\n' "$send_model" "$send_provider" | ||
| printf '**Prompt:** %s\n\n' "$send_prompt" | ||
| printf '**Reply:**\n\n```\n' | ||
| jq -r '.reply' "$artifact_dir/console-send.json" 2>/dev/null | ||
| printf '```\n\n' | ||
| fi | ||
|
|
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Guard the console-message evidence block against a failed/empty console-send.json.
console_send.py's stdout is redirected straight to console-send.json (Line 512) even when the process exits nonzero before ever calling json.dump. When that happens this block still renders a "Console message check" heading with an empty/garbled reply (the jq -r '.reply' error is swallowed by 2>/dev/null), which reads as a partially-successful check even though message_check was set to failed.
♻️ Suggested fix
- if [[ -f "$artifact_dir/console-send.json" ]]; then
+ if [[ -s "$artifact_dir/console-send.json" ]] && jq -e . "$artifact_dir/console-send.json" >/dev/null 2>&1; then
printf '## Console message check (%s via %s)\n\n' "$send_model" "$send_provider"
printf '**Prompt:** %s\n\n' "$send_prompt"
printf '**Reply:**\n\n```\n'
jq -r '.reply' "$artifact_dir/console-send.json" 2>/dev/null
printf '```\n\n'
fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if [[ -f "$artifact_dir/console-send.json" ]]; then | |
| printf '## Console message check (%s via %s)\n\n' "$send_model" "$send_provider" | |
| printf '**Prompt:** %s\n\n' "$send_prompt" | |
| printf '**Reply:**\n\n```\n' | |
| jq -r '.reply' "$artifact_dir/console-send.json" 2>/dev/null | |
| printf '```\n\n' | |
| fi | |
| if [[ -s "$artifact_dir/console-send.json" ]] && jq -e . "$artifact_dir/console-send.json" >/dev/null 2>&1; then | |
| printf '## Console message check (%s via %s)\n\n' "$send_model" "$send_provider" | |
| printf '**Prompt:** %s\n\n' "$send_prompt" | |
| printf '**Reply:**\n\n |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@harness/tests/quickstart/run-ci.sh` around lines 203 - 210, Update the
console-message evidence block in run-ci.sh to render only when
console-send.json contains valid JSON with a usable reply, rather than merely
when the file exists. Use jq validation or an equivalent guard before printing
the heading and reply, while preserving the existing output for successful
console_send.py runs and avoiding suppressed jq errors being presented as
evidence.
Comment on lines
+489
to
+525
| # --------------------------------------------------------------------------- | ||
| # Steps 7-8: add the Z.AI provider and prove a real model reply through the | ||
| # Console. Gated on ZAI_API_KEY: the engine (started in Step 2) inherited | ||
| # this shell's environment, so the provider worker it spawns sees the key. | ||
| # --------------------------------------------------------------------------- | ||
| if [[ -n "${ZAI_API_KEY:-}" ]]; then | ||
| stage_start worker_add_provider_zai | ||
| log "Step 7: iii worker add provider-zai" | ||
| run_add provider-zai 2>&1 | tee "$log_dir/provider-add.log" | ||
| wait_for_function provider::zai::stream | ||
| stage_end | ||
|
|
||
| stage_start console_message_check | ||
| log "Step 8: send a message through the Console /ws proxy (model=$send_model provider=$send_provider)" | ||
| message_check="failed" | ||
| python3 -m venv "$run_root/venv" >>"$log_dir/console-send.log" 2>&1 \ | ||
| || die "python3 -m venv failed (see console-send.log)" | ||
| "$run_root/venv/bin/pip" install --quiet websockets >>"$log_dir/console-send.log" 2>&1 \ | ||
| || die "pip install websockets failed (see console-send.log)" | ||
| if ! "$run_root/venv/bin/python" "$script_dir/console_send.py" \ | ||
| --url "ws://127.0.0.1:$console_port/ws" \ | ||
| --model "$send_model" --provider "$send_provider" \ | ||
| --prompt "$send_prompt" --timeout "$turn_timeout_seconds" \ | ||
| >"$artifact_dir/console-send.json" \ | ||
| 2> >(tee -a "$log_dir/console-send.log" >&2); then | ||
| die "console message send failed (see console-send.log)" | ||
| fi | ||
| reply=$(jq -er '.reply | select(length > 0)' "$artifact_dir/console-send.json") \ | ||
| || die "console send did not produce a non-empty assistant reply" | ||
| ok "assistant replied through the Console (${#reply} chars)" | ||
| message_check="passed" | ||
| stage_end | ||
| else | ||
| log "Steps 7-8: ZAI_API_KEY is not set; skipping provider add + Console message check" | ||
| fi | ||
|
|
||
| log "ALL QUICKSTART ASSERTIONS PASSED ($cli_version, channel=$channel, message_check=$message_check)" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Process-substitution stderr tee can race with die on failure.
2> >(tee -a "$log_dir/console-send.log" >&2) (Line 513) writes stderr asynchronously via a subshell that bash does not wait on. If the python command fails and die exits right after, the last lines of console-send.log — the exact diagnostics needed to debug the failure — can be missing when collect_evidence/artifact upload runs.
♻️ Suggested fix (avoid async process substitution)
- if ! "$run_root/venv/bin/python" "$script_dir/console_send.py" \
- --url "ws://127.0.0.1:$console_port/ws" \
- --model "$send_model" --provider "$send_provider" \
- --prompt "$send_prompt" --timeout "$turn_timeout_seconds" \
- >"$artifact_dir/console-send.json" \
- 2> >(tee -a "$log_dir/console-send.log" >&2); then
- die "console message send failed (see console-send.log)"
- fi
+ if ! "$run_root/venv/bin/python" "$script_dir/console_send.py" \
+ --url "ws://127.0.0.1:$console_port/ws" \
+ --model "$send_model" --provider "$send_provider" \
+ --prompt "$send_prompt" --timeout "$turn_timeout_seconds" \
+ >"$artifact_dir/console-send.json" \
+ 2>>"$log_dir/console-send.log"; then
+ cat "$log_dir/console-send.log" >&2
+ die "console message send failed (see console-send.log)"
+ fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| # --------------------------------------------------------------------------- | |
| # Steps 7-8: add the Z.AI provider and prove a real model reply through the | |
| # Console. Gated on ZAI_API_KEY: the engine (started in Step 2) inherited | |
| # this shell's environment, so the provider worker it spawns sees the key. | |
| # --------------------------------------------------------------------------- | |
| if [[ -n "${ZAI_API_KEY:-}" ]]; then | |
| stage_start worker_add_provider_zai | |
| log "Step 7: iii worker add provider-zai" | |
| run_add provider-zai 2>&1 | tee "$log_dir/provider-add.log" | |
| wait_for_function provider::zai::stream | |
| stage_end | |
| stage_start console_message_check | |
| log "Step 8: send a message through the Console /ws proxy (model=$send_model provider=$send_provider)" | |
| message_check="failed" | |
| python3 -m venv "$run_root/venv" >>"$log_dir/console-send.log" 2>&1 \ | |
| || die "python3 -m venv failed (see console-send.log)" | |
| "$run_root/venv/bin/pip" install --quiet websockets >>"$log_dir/console-send.log" 2>&1 \ | |
| || die "pip install websockets failed (see console-send.log)" | |
| if ! "$run_root/venv/bin/python" "$script_dir/console_send.py" \ | |
| --url "ws://127.0.0.1:$console_port/ws" \ | |
| --model "$send_model" --provider "$send_provider" \ | |
| --prompt "$send_prompt" --timeout "$turn_timeout_seconds" \ | |
| >"$artifact_dir/console-send.json" \ | |
| 2> >(tee -a "$log_dir/console-send.log" >&2); then | |
| die "console message send failed (see console-send.log)" | |
| fi | |
| reply=$(jq -er '.reply | select(length > 0)' "$artifact_dir/console-send.json") \ | |
| || die "console send did not produce a non-empty assistant reply" | |
| ok "assistant replied through the Console (${#reply} chars)" | |
| message_check="passed" | |
| stage_end | |
| else | |
| log "Steps 7-8: ZAI_API_KEY is not set; skipping provider add + Console message check" | |
| fi | |
| log "ALL QUICKSTART ASSERTIONS PASSED ($cli_version, channel=$channel, message_check=$message_check)" | |
| # --------------------------------------------------------------------------- | |
| # Steps 7-8: add the Z.AI provider and prove a real model reply through the | |
| # Console. Gated on ZAI_API_KEY: the engine (started in Step 2) inherited | |
| # this shell's environment, so the provider worker it spawns sees the key. | |
| # --------------------------------------------------------------------------- | |
| if [[ -n "${ZAI_API_KEY:-}" ]]; then | |
| stage_start worker_add_provider_zai | |
| log "Step 7: iii worker add provider-zai" | |
| run_add provider-zai 2>&1 | tee "$log_dir/provider-add.log" | |
| wait_for_function provider::zai::stream | |
| stage_end | |
| stage_start console_message_check | |
| log "Step 8: send a message through the Console /ws proxy (model=$send_model provider=$send_provider)" | |
| message_check="failed" | |
| python3 -m venv "$run_root/venv" >>"$log_dir/console-send.log" 2>&1 \ | |
| || die "python3 -m venv failed (see console-send.log)" | |
| "$run_root/venv/bin/pip" install --quiet websockets >>"$log_dir/console-send.log" 2>&1 \ | |
| || die "pip install websockets failed (see console-send.log)" | |
| if ! "$run_root/venv/bin/python" "$script_dir/console_send.py" \ | |
| --url "ws://127.0.0.1:$console_port/ws" \ | |
| --model "$send_model" --provider "$send_provider" \ | |
| --prompt "$send_prompt" --timeout "$turn_timeout_seconds" \ | |
| >"$artifact_dir/console-send.json" \ | |
| 2>>"$log_dir/console-send.log"; then | |
| cat "$log_dir/console-send.log" >&2 | |
| die "console message send failed (see console-send.log)" | |
| fi | |
| reply=$(jq -er '.reply | select(length > 0)' "$artifact_dir/console-send.json") \ | |
| || die "console send did not produce a non-empty assistant reply" | |
| ok "assistant replied through the Console (${`#reply`} chars)" | |
| message_check="passed" | |
| stage_end | |
| else | |
| log "Steps 7-8: ZAI_API_KEY is not set; skipping provider add + Console message check" | |
| fi | |
| log "ALL QUICKSTART ASSERTIONS PASSED ($cli_version, channel=$channel, message_check=$message_check)" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@harness/tests/quickstart/run-ci.sh` around lines 489 - 525, Replace the
asynchronous process substitution on the console_send.py stderr redirection in
the console_message_check stage with a synchronous logging approach that
preserves stderr output in console-send.log before the failure path calls die.
Keep stderr visible to the terminal and ensure the complete diagnostics are
available for collect_evidence and artifact upload.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the quickstart validator beyond install/boot: it now proves that a message sent through the Console gets a real model reply, using z.ai as the provider — plus an option to validate the
nextinstaller channel.Live message check (Steps 7–8, gated on
ZAI_API_KEY)iii worker add provider-zai) and waits forprovider::zai::streamto register.console_send.py, which speaks the engine WebSocket protocol (invokefunction/invocationresult) through the Console's/wsproxy — the exact path the browser SPA uses, so a pass exercises Console proxy → engine → harness → llm-router → provider-zai end to end:harness::send{message, model: glm-5.2, provider: zai}(bounded:max_turns: 2,max_output_tokens: 1024,max_total_tokens: 16384)harness::statusuntil the turn completes (fails fast onfailed/cancelled)session::messages→ assert a non-empty assistant replyEVIDENCE.md(andconsole-send.jsonin the result artifact), so every run shows the actual model output.result.json/EVIDENCE.md— the install/boot validation still runs.ZAI_API_KEY; the key never lands in config or artifacts.nextchannel optionIII_CHANNEL=nextmakes the installer run with--next; exposed as achannelchoice input onworkflow_dispatch(cron stays onmain). Concurrency group and job name are per-channel.Evidence cleanup
EVIDENCE.md: theiii.locksection and thediscovery.log/console-status.logtails (poll chatter). Both logs still ship in theharness-quickstart-logsartifact.Testing
bash -n,python3 -m py_compile, and YAML parse pass.rbac-proxy/src/interceptor.rs,SendRequest/SendOptionsfromharness/src/functions/send.rs, terminal-turn logic fromharness/tests/e2e/src/context.rs, transcript pagination from the same, message shape fromharness/tests/e2e/src/scenarios/common.rs.workflow_dispatchrun on this branch validates end to end (uses the repoZAI_API_KEYsecret already consumed by the Harness E2E nightly).Label
no-ticket: CI/validator tooling only.Summary by CodeRabbit
New Features
mainornextinstallation channel.Bug Fixes
Documentation
nextchannel usage.