Skip to content

Conversation

@jmcarp
Copy link
Contributor

@jmcarp jmcarp commented Oct 27, 2025

Apply a few ddl-only optimizations to the oximeter measurements tables:

  • Encode timestamps columns using delta encoding. Rows are sorted by (timeseries_name, timeseries_key, timestamp), so timestamps compress very well as deltas.
  • Partition measurements tables by timestamp, within days. Many queries care about a small time range, and partitioning by day will make those queries significantly cheaper.

Note: this is a draft, since I need to better understand how clickhouse migrations work, and how to do this safely. I'll apply these changes to all measurements tables if/when we agree to an approach and convince ourselves that this will be safe to run.

@jmcarp jmcarp requested a review from bnaecker October 27, 2025 19:05
@jmcarp
Copy link
Contributor Author

jmcarp commented Oct 28, 2025

I dug into this a bit more, pulling a few known slow queries (to make sure they got faster) and a known fast query (to make sure it didn't get slower). On closer inspection, partitioning seems pretty useful, speeding up the slower queries by over 50%, but changing the timestamp encoding actually makes us go slower. I'll update this PR to add partitioning and leave encoding alone.

  • virtual_machine:vcpu_usage: ~0.515s | ~0.18s | ~0.5s
  • hardware_component:current: ~1.8s | ~0.52 | ~0.66s
  • switch_data_link:fec_high_symbol_errors : ~0.03s | 0.02s

Note: I talked with @bnaecker about this, and if we decide to go ahead with the partitioning change, we'll need to wait on #7488. Until that issue is resolved, we effectively have no good way to update clickhouse schemas.

Partition measurements tables by timestamp, within days. Many queries care
about a small time range, and partitioning by day will make those queries
significantly cheaper.
@jmcarp jmcarp force-pushed the jmcarp/oximeter-optimize-measurements branch from 1a540ef to f416ee0 Compare October 28, 2025 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants