feat: observability (token usage, run log, acceptance metrics) - #66
feat: observability (token usage, run log, acceptance metrics)#66Benkapner wants to merge 3 commits into
Conversation
Users have no idea what a run costs. With parallel generation plus verification, a large PR fans out to dozens of LLM calls invisibly. Add a UsageTracker that accumulates prompt/completion tokens per call tagged by stage, and appends a collapsed token usage summary to the update-mode confirmation comment. Optional cost-per-1m-input and cost-per-1m-output action inputs enable estimated cost reporting; when unset, only token counts are shown. Backends that omit usage show "not reported" rather than zero.
Debugging a bad generation currently means reading interleaved print() output from parallel threads, with prompts unrecoverable. Add a JSONL run log with one record per LLM call: timestamp, stage, file path, prompt/response length, token usage, latency, outcome. Behind an opt-in debug-artifacts input, include full prompt and response text. All output is routed through sanitize_output() so credentials cannot leak into an artifact.
The checkbox UI already generates the project's quality signal: how many suggested files a human leaves checked. On [update-docs], parse the prior review comment's checkbox state and compute suggested vs accepted. Emit the ratio into the run output and as an Action output. Does not send anything off-repo.
|
🤖 Finished Review · ✅ Success · Started 5:35 AM UTC · Completed 5:53 AM UTC Commit: |
ReviewFindingsHigh
Medium
Low
Labels: PR adds observability features (token tracking, cost estimation, run logs) to the GitHub Action Next steps:
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
| @@ -81,6 +93,8 @@ outputs: | |||
| description: 'JSON array of modified files' | |||
| pr-created: | |||
| description: 'Whether a PR was created' | |||
There was a problem hiding this comment.
[high] api-contract
The action declares an acceptance-rate output but entrypoint.sh never writes acceptance-rate=... to $GITHUB_OUTPUT. The acceptance rate is only printed to stdout in suggest_docs.py. Downstream workflows referencing steps..outputs.acceptance-rate will always get an empty string.
Suggested fix: Write the acceptance rate to GITHUB_OUTPUT in suggest_docs.py (e.g., append acceptance-rate= to os.environ.get('GITHUB_OUTPUT')).
| cost_input = os.environ.get("COST_PER_1M_INPUT", "") | ||
| cost_output = os.environ.get("COST_PER_1M_OUTPUT", "") | ||
| usage_tracker = UsageTracker( | ||
| cost_per_1m_input=float(cost_input) if cost_input else None, |
There was a problem hiding this comment.
[medium] error-handling-gap
float(cost_input) will crash with ValueError on non-numeric input (e.g., '$3.00'). The existing get_max_context_chars() in config.py uses try/except with warning and fallback.
Suggested fix: Wrap each float() call in try/except ValueError, log a warning, and default to None.
|
|
||
| if self._cost_input and self._cost_output and all_reported: | ||
| cost = (total_prompt / 1_000_000) * self._cost_input + ( | ||
| total_completion / 1_000_000 |
There was a problem hiding this comment.
[low] logic-error
Total row in format_summary() shows all calls but only sums tokens from stages that reported usage, making totals misleading when some stages lack usage data.
| ) * self._cost_output | ||
| lines.append(f"\nEstimated cost: ${cost:.4f}") | ||
|
|
||
| table = "\n".join(lines) |
There was a problem hiding this comment.
[low] edge-case
Cost condition uses truthiness (if self._cost_input and ...), so a cost of 0.0 suppresses the cost line. Use 'is not None' for precision.
fullsend-ai-review
Bot
added
feature
python
1 participant
Summary
Adds instrumentation so users can see what the tool did, what it cost, and whether its suggestions are useful. Generated content is unchanged.
src/telemetry.py): thread-safe accumulator for prompt/completion tokens per stage. Collapsed<details>block appended to the update-mode confirmation comment. Optionalcost-per-1m-input/cost-per-1m-outputinputs for estimated cost; when unset, shows token counts only. Backends without usage data show "not reported".src/run_log.py): JSONL with one record per LLM call (timestamp, stage, file, prompt/response length, usage, latency, outcome). Full prompt/response text behind opt-indebug-artifactsinput. Sanitized throughsanitize_output().[update-docs], computes suggested vs accepted from the prior review's checkbox state. Emits to run output and as anacceptance-rateAction output. Local only.Test plan
uv run pytest -vpasses (428 tests)uv run ruff check src/ tests/anduv run ruff format --check src/ tests/clean