You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
CI performance is a cluster of related questions, not one skill, and the pieces currently sit in three different states.
Adjacent work, and its real status, since the three differ:
e2e-flakiness-patterns — shipped. A catalogue of what flakiness looks like, not a measurement: it does not say whether this repository has it, where, or whether it is getting worse.
G6 CI job-duration delta — an evidence lane, in review, not yet on the default branch. Prices a known change against CI.
What is missing is everything between them. Nobody can currently answer, with evidence rather than impression: which jobs dominate wall-clock and how that has moved; which tests are flaky ranked by cost rather than by annoyance; how much of the queue is retry; whether a given week's slowdown is the runner fleet, a dependency, or a change we made. These are asked repeatedly and answered anecdotally.
The reason to treat this as a cluster rather than one skill is that the questions have different shapes. Ranking flaky tests is an aggregation over history. Attributing a slowdown is a search. Pricing a known change is a paired comparison. One skill covering all three would be three protocols wearing a trench coat, and each would be weaker for it.
What the data actually supports
Inspecting the CI Health dashboard answers what the cluster can be built on, and the answer is that most of it already exists.
The dashboard does not read DevLake. Its panels query a PostgreSQL source, against sonarcloud_metrics.github_pr_runs and sibling tables automated_test_metrics, metrics, and segment_events_coverage_metrics. Five sources are wired into the instance overall: that PostgreSQL store, a DevLake MySQL store, Prometheus, Google Sheets, and Sentry.
github_pr_runs already carries the columns the cluster needs — run_attempt, run_execution_time_minutes, run_conclusion, workflow_name, run_url, project, pr_merged_at. Retry rate is not merely derivable but already computed in a shipped panel, as the share of runs with run_attempt above one. So per-job wall-clock ranking and retry-cost ranking are queries against an existing table, not a data-modelling project.
Two consequences worth carrying into design. Sentry is a datasource on the same instance, so correlating production symptoms against CI history is a two-source query in one place rather than a stitch across two APIs — which simplifies the regression-attribution work considerably. And the existing dashboard is oriented at test counts, coverage, and success rate; wall-clock ranking, retry share of queue, and flakiness ranked by cost are not on it, which is precisely the gap this cluster fills.
Goal
Decide the cluster's decomposition, then build it — a small set of skills over CI history that answer the recurring questions with reproducible queries rather than dashboard screenshots.
Requirements
Every figure traces to a query a reader can re-run. A dashboard screenshot is not evidence, because it cannot be checked and cannot be diffed against next month.
Flakiness is ranked by cost — retry minutes and blocked pull requests — not by failure count. A test that fails often but reruns in twenty seconds matters less than one that fails rarely and costs a full pipeline.
Slowdown findings distinguish our changes from fleet and dependency effects, and say which when the evidence supports it and say so when it does not.
Findings state what would have to be true for the correlation to be coincidental, and whether that holds across prior comparable periods.
Access
Read access to the metrics store is the one unresolved dependency, and it is narrower than a network or infrastructure problem. The host is reachable from a build machine and its health endpoint responds; unauthenticated data endpoints return 401 and the dashboard redirects to a login. So what is required is a durable read credential, not connectivity work.
One-off human access is not a working arrangement. A skill must be scriptable, re-runnable in CI, and usable by someone who is not the person who authenticated — so anything resting on interactive login is unusable by the automation it exists to serve.
Resolutions, in order of preference: a read-scoped Grafana service account token; direct read access to the PostgreSQL metrics store, bypassing the dashboard layer entirely, since the tables above are the real dependency and the dashboard is only a view over them; or an existing internal interface that already fronts this data.
Related to but distinct from the toolchain-benchmarking work, which prices a known change and needs no historical access. This cluster reasons over CI history, which is why it is gated on the data and that one is not.
Context
CI performance is a cluster of related questions, not one skill, and the pieces currently sit in three different states.
Adjacent work, and its real status, since the three differ:
e2e-flakiness-patterns— shipped. A catalogue of what flakiness looks like, not a measurement: it does not say whether this repository has it, where, or whether it is getting worse.G6 CI job-duration delta— an evidence lane, in review, not yet on the default branch. Prices a known change against CI.What is missing is everything between them. Nobody can currently answer, with evidence rather than impression: which jobs dominate wall-clock and how that has moved; which tests are flaky ranked by cost rather than by annoyance; how much of the queue is retry; whether a given week's slowdown is the runner fleet, a dependency, or a change we made. These are asked repeatedly and answered anecdotally.
The reason to treat this as a cluster rather than one skill is that the questions have different shapes. Ranking flaky tests is an aggregation over history. Attributing a slowdown is a search. Pricing a known change is a paired comparison. One skill covering all three would be three protocols wearing a trench coat, and each would be weaker for it.
What the data actually supports
Inspecting the CI Health dashboard answers what the cluster can be built on, and the answer is that most of it already exists.
The dashboard does not read DevLake. Its panels query a PostgreSQL source, against
sonarcloud_metrics.github_pr_runsand sibling tablesautomated_test_metrics,metrics, andsegment_events_coverage_metrics. Five sources are wired into the instance overall: that PostgreSQL store, a DevLake MySQL store, Prometheus, Google Sheets, and Sentry.github_pr_runsalready carries the columns the cluster needs —run_attempt,run_execution_time_minutes,run_conclusion,workflow_name,run_url,project,pr_merged_at. Retry rate is not merely derivable but already computed in a shipped panel, as the share of runs withrun_attemptabove one. So per-job wall-clock ranking and retry-cost ranking are queries against an existing table, not a data-modelling project.Two consequences worth carrying into design. Sentry is a datasource on the same instance, so correlating production symptoms against CI history is a two-source query in one place rather than a stitch across two APIs — which simplifies the regression-attribution work considerably. And the existing dashboard is oriented at test counts, coverage, and success rate; wall-clock ranking, retry share of queue, and flakiness ranked by cost are not on it, which is precisely the gap this cluster fills.
Goal
Decide the cluster's decomposition, then build it — a small set of skills over CI history that answer the recurring questions with reproducible queries rather than dashboard screenshots.
Requirements
Every figure traces to a query a reader can re-run. A dashboard screenshot is not evidence, because it cannot be checked and cannot be diffed against next month.
Flakiness is ranked by cost — retry minutes and blocked pull requests — not by failure count. A test that fails often but reruns in twenty seconds matters less than one that fails rarely and costs a full pipeline.
Slowdown findings distinguish our changes from fleet and dependency effects, and say which when the evidence supports it and say so when it does not.
Findings state what would have to be true for the correlation to be coincidental, and whether that holds across prior comparable periods.
Access
Read access to the metrics store is the one unresolved dependency, and it is narrower than a network or infrastructure problem. The host is reachable from a build machine and its health endpoint responds; unauthenticated data endpoints return 401 and the dashboard redirects to a login. So what is required is a durable read credential, not connectivity work.
One-off human access is not a working arrangement. A skill must be scriptable, re-runnable in CI, and usable by someone who is not the person who authenticated — so anything resting on interactive login is unusable by the automation it exists to serve.
Resolutions, in order of preference: a read-scoped Grafana service account token; direct read access to the PostgreSQL metrics store, bypassing the dashboard layer entirely, since the tables above are the real dependency and the dashboard is only a view over them; or an existing internal interface that already fronts this data.
Notes
Tracked in the skills repository as #105.
Related to but distinct from the toolchain-benchmarking work, which prices a known change and needs no historical access. This cluster reasons over CI history, which is why it is gated on the data and that one is not.
Tracked in Jira as
AIENG-96.