Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 32 additions & 6 deletions .github/workflows/harness-quickstart.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,20 +25,44 @@ jobs:
HARNESS_QUICKSTART_ARTIFACTS_DIR: ${{ github.workspace }}/target/harness-quickstart
run: harness/tests/quickstart/run-ci.sh

- name: Show quickstart logs
if: always()
run: |
shopt -s nullglob
if [[ -f target/harness-quickstart/EVIDENCE.md ]]; then
echo "::group::EVIDENCE.md"
cat target/harness-quickstart/EVIDENCE.md
echo "::endgroup::"
fi
logs=(target/harness-quickstart/logs/*.log)
if ((${#logs[@]} == 0)); then
echo "No quickstart logs were produced."
exit 0
fi
for log in "${logs[@]}"; do
echo "::group::$(basename "$log")"
cat "$log"
echo "::endgroup::"
done

- name: Write quickstart summary
if: always()
run: |
{
echo "### Harness quickstart"
echo
if [[ -f target/harness-quickstart/result.json ]]; then
if [[ -f target/harness-quickstart/EVIDENCE.md ]]; then
cat target/harness-quickstart/EVIDENCE.md
elif [[ -f target/harness-quickstart/result.json ]]; then
echo "### Harness quickstart"
echo
jq -r '
"| Status | CLI | Duration |",
"| --- | --- | ---: |",
"| \(.status) | \(.cli_version) | \(.elapsed_ms) ms |",
(if .failure_reason != "" then "", "Failure: \(.failure_reason)" else empty end)
' target/harness-quickstart/result.json
else
echo "### Harness quickstart"
echo
echo "No quickstart result was produced."
fi
} >> "$GITHUB_STEP_SUMMARY"
Expand All @@ -50,18 +74,20 @@ jobs:
name: harness-quickstart-result
path: |
target/harness-quickstart/result.json
target/harness-quickstart/EVIDENCE.md
target/harness-quickstart/timings.tsv
target/harness-quickstart/cli-version.txt
target/harness-quickstart/console-status.json
target/harness-quickstart/config.yaml
target/harness-quickstart/iii.lock
retention-days: 7
if-no-files-found: ignore

- name: Upload quickstart diagnostics
if: failure()
- name: Upload quickstart logs
if: always()
uses: actions/upload-artifact@v6
with:
name: harness-quickstart-diagnostics
name: harness-quickstart-logs
path: |
target/harness-quickstart/logs/
target/harness-quickstart/console.html
Expand Down
8 changes: 8 additions & 0 deletions harness/tests/quickstart/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,14 @@ make -C harness quickstart-validate
The local machine needs `curl` and `jq`; CI provides both on the standard
Ubuntu runner.

The run narrates every step with timestamped log lines and `[ok]` assertions,
records per-stage durations in `timings.tsv` (rendered as a `[timing breakdown]`
at the end and embedded in `result.json`), and always writes an `EVIDENCE.md`
digest — status, CLI version, timing, and the tail of every log — into the
artifacts directory, on success and on failure alike. In CI the evidence file
becomes the job step summary and the raw logs are printed in collapsible
groups, so a run can be audited without downloading anything.

The CI workflow runs this check nightly and manually against the `latest`
registry artifacts. It intentionally does not test model generation; provider
credentials and model quality are covered by the Harness E2E workflow.
Loading
Loading