An open benchmark for real-time analytics cost-performance across the complete analytics path.
CostBench measures the work and cost required to make continuously arriving data query-ready and serve analytical queries over it. The current full-path quotes study includes accepted runs for ClickHouse Cloud, Snowflake, Google BigQuery, and Amazon Redshift Serverless.
Note
A static query benchmark starts after data has been loaded and prepared. CostBench also measures continuous ingest, maintenance of query-ready structures, freshness, and query serving while that work remains active.
| Area | Purpose |
|---|---|
| Full-path benchmarks | Current end-to-end real-time methodology and workloads |
| Quotes benchmark | Accepted multi-provider study, evidence map, and reproduction order |
| Global visualizations | Provider-neutral chart manifest and reproducible renderers |
| Legacy query benchmark | Read-side comparison over already-prepared data |
The benchmark keeps the analytics path live from source to answer:
- Events arrive continuously at a fixed target rate.
- The provider writes those events into its raw-data path.
- The raw layout remains usable for drill-down queries.
- A derived aggregate is maintained for dashboard queries.
- Dashboard and drill-down workloads run while ingest and maintenance continue.
The published evidence covers:
- ingest progress and successful row counts;
- raw and aggregate query latency during active ingestion;
- persisted materialized-view freshness;
- complete fresh-data-path and matched query cost;
- provider configuration and pricing assumptions;
- source JSONL, reconciled windows, generated CSV, SVG, PNG, and provenance summaries.
This is not a bulk-load benchmark. Systems are evaluated as continuously operating real-time analytics paths, including provider-specific components such as background refresh compute, serverless ingestion services, or a required broker layer.
| System | Accepted evidence | Comparison role |
|---|---|---|
| ClickHouse Cloud | results_t2/ |
Pairwise reference and full-path baseline |
| Snowflake | results/t2/ |
Accepted Run14 with normalized mixed-rate query attribution |
| BigQuery | bq-full-t2-20260810_152224/ |
Accepted T2 with Capacity and On-demand alternatives |
| Redshift Serverless | results/t2/ |
Accepted T2 with SUPER and typed read alternatives |
The global score is:
(complete fresh-data-path cost + matched query cost) × accumulated query runtime
Lower is better. Each non-ClickHouse score is normalized within its own accepted pairwise row-progress window. The global chart combines those accepted pairwise ratios; it does not claim a single cross-provider iteration join. See the quotes methodology and the generated provenance JSON beside every chart for the exact contract.
CostBench publishes the scripts and evidence needed to inspect benchmark claims:
- workload, schema, and query definitions;
- ingest and fixed-rate runner implementations;
- provider configuration and pricing files;
- raw runner results and row-progress reconciliation reports;
- cost calculations and accepted summaries;
- fail-closed visualization manifests and slide-ready outputs.
Generated summaries store repository-relative source paths and SHA-256 hashes. Credential files are
local-only: the repository ignores every *_credentials.txt path, and CI rejects credential
artifacts or high-confidence secret material if either is staged accidentally.
The full-path methodology builds on two earlier CostBench studies:
- How the 5 major cloud data warehouses compare on cost-performance — read-side cost-performance at 1B, 10B, and 100B rows.
- Agentic analytics starts with query-ready data — the write-side cost of keeping raw data query-ready.
Current methodology and background:
- Introducing CostBench
- The end-to-end cost-performance of real-time analytics
- How the 5 major cloud data warehouses really bill you
Cost-performance claims should be reviewable. Pull requests that improve a configuration, pricing
assumption, cost boundary, reconciliation rule, or disclosure are welcome. Keep secrets outside the
repository and run python3 scripts/check_repository_hygiene.py before committing.
See LICENSE.