Label forecast output as AI Credits and cache AIC across runs - #48797
Merged
Conversation
Make `gh aw forecast` output clearly denote AI Credits (AIC) and reuse cached AIC values across repeated runs. Output clarity: - Table headers now carry units: P50 AIC/Run, P95 AIC/Run, Weekly AIC (P50), Monthly AIC (P50). - Added an "All numeric figures are AI Credits (AIC)" header line and an "AIC = AI Credits" prefix on the explanatory footnote. Caching: - Add a dedicated, CLI-version-gated forecast_aic.json per-run cache so repeated forecast runs reuse computed AIC without re-scanning the run directory or re-parsing the usage artifact. - loadCachedRunAIC now checks run_summary.json -> forecast_aic.json -> download, and persists the computed AIC after a download. A dedicated file avoids corrupting the shared run_summary.json "fully processed" marker used by `gh aw logs`/`audit`. - Cache hits/misses/writes are logged via forecastRunLog (enable with the DEBUG env var, e.g. DEBUG=cli:forecast_run). Also document forecast_aic.json in the artifact description map and the forecast specification (R-SAMP-013), and add unit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Clarifies forecast units and adds reusable per-run AI Credits caching.
Changes:
- Labels forecast cost projections as AIC.
- Adds version-gated
forecast_aic.jsoncaching. - Documents and tests the cache format.
Show a summary per file
| File | Description |
|---|---|
pkg/cli/forecast_types.go |
Adds AIC units to table headers. |
pkg/cli/forecast_render.go |
Clarifies forecast output units. |
pkg/cli/forecast_compute.go |
Integrates forecast-cache reads and writes. |
pkg/cli/forecast_cache.go |
Implements the AIC cache. |
pkg/cli/forecast_cache_test.go |
Tests cache validation and persistence. |
pkg/cli/audit_report.go |
Describes the cache in audit output. |
docs/src/content/docs/specs/forecast-specification.md |
Documents forecast caching requirements. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Medium
| forecastRunLog.Printf("AIC from usage artifact for run %d: aic=%.3f", runID, tokenUsage.TotalAIC) | ||
| // Persist the computed AIC so subsequent forecast runs hit the fast forecast cache | ||
| // instead of re-scanning the directory and re-parsing the usage artifact. | ||
| saveForecastAICCache(dir, runID, tokenUsage.TotalAIC) |
Contributor
There was a problem hiding this comment.
Fixed in 4536c1a by isolating the forecast cache tests in a temporary working directory in pkg/cli/forecast_test.go.
| fmt.Fprintln(os.Stderr, console.FormatInfoMessage( | ||
| fmt.Sprintf("Workflow Forecast — weekly & monthly projections (based on last %d days of history)", config.Days))) | ||
| fmt.Fprintln(os.Stderr, console.FormatInfoMessage( | ||
| "All numeric figures are AI Credits (AIC) — the gh-aw cost metric.")) |
Contributor
There was a problem hiding this comment.
Fixed in 4536c1a by narrowing the banner text in pkg/cli/forecast_render.go to cost/projection figures.
Collaborator
Author
Collaborator
Author
|
@copilot run pr-finisher skill |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Contributor
Ran it. Ready for human merge pending CI re-trigger. |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
|
🎉 This pull request is included in a new release. Release: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two improvements to
gh aw forecast:1. Make it clear the figures are AI Credits (AIC)
The forecast table previously showed bare numbers (
P50/Run,Weekly (P50), …) with no unit. Now:P50 AIC/Run,P95 AIC/Run,Weekly AIC (P50),Monthly AIC (P50).2. Maximize cache reuse across repeated forecast runs
Runs first seen by
forecast(not bygh aw logs) had norun_summary.json, so every repeat re-scanned the run directory and re-parsed the usage artifact. Writing a partialrun_summary.jsonwas unsafe — it's the "fully processed" marker forgh aw logs/audit. Instead:forecast_aic.jsonper-run cache (pkg/cli/forecast_cache.go).loadCachedRunAICnow checksrun_summary.json→forecast_aic.json→ download, and persists the computed AIC after a download.forecastRunLog, enabled through theDEBUGenv var (e.g.DEBUG=cli:forecast_run,DEBUG=*, orACTIONS_RUNNER_DEBUG=truein the runner).Verification
DEBUG=cli:forecast_run gh aw forecast smoke-copilot:forecast_aic.jsonfor each sampled run.Build,
go vet, and forecast tests pass. Added unit tests (forecast_cache_test.go) and documented the file inaudit_report.goand the forecast spec (R-SAMP-013). Cache files live under the gitignored.github/aw/logs/.