Skip to content

Conversation

@yeya24
Copy link
Contributor

@yeya24 yeya24 commented Oct 23, 2025

What this PR does:

This PR adds similar functionality of lazy postings from Store Gateway but to Ingester.

We have found that in Ingester, low selectivity matcher like =~".+" most of the case cannot filter out anything but can be extremely expensive as it requires fetching postings for all values for the label and intersect with other postings.

Here is one example CPU profile where we spend 50% of CPU on PostingsForAllLabelValues which is used to evaluate =~.+ matcher.

image

Because of the low selectivity of such matcher, it seems cheaper to apply the label matcher lazily on the selected series set instead. This PR adds a new config enable_matcher_optimization to ingester to enable this matcher optimization.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@SungJin1212
Copy link
Member

It looks promising. Can you share some benchmarks?

Comment on lines +65 to +74
"no optimization needed with .* regex returns all as select": {
input: []*labels.Matcher{
labels.MustNewMatcher(labels.MatchRegexp, "job", ".*"),
labels.MustNewMatcher(labels.MatchEqual, "instance", "server1"),
},
expectedSelect: []*labels.Matcher{
labels.MustNewMatcher(labels.MatchRegexp, "job", ".*"),
labels.MustNewMatcher(labels.MatchEqual, "instance", "server1"),
},
expectedLazy: nil,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, do we need to keep the .*?

Copy link
Contributor Author

@yeya24 yeya24 Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can keep .* as PostingsForMatcher will anyway ignore it.

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Oct 24, 2025
@yeya24
Copy link
Contributor Author

yeya24 commented Oct 24, 2025

@SungJin1212 Let me add a benchmark test and share the results

@yeya24
Copy link
Contributor Author

yeya24 commented Oct 24, 2025

Benchmark result. For each test case I ran it with and without the optimization.

For the sparse label test case, performance becomes worse but that's expected and rare. For most of the cases if there is no empty value then this optimizations helps.

-------- BEGIN BENCHMARK --------
goos: darwin
goarch: arm64
pkg: github.com/cortexproject/cortex/pkg/ingester
cpu: Apple M1 Pro
BenchmarkIngester_QueryStreamChunks_MatcherOptimization/complex_matchers_optimization_disabled-10             21490           1098345 ns/op         1279912 B/op         13058 allocs/op
BenchmarkIngester_QueryStreamChunks_MatcherOptimization/complex_matchers_optimization_enabled-10              27031            888221 ns/op         1186002 B/op         13038 allocs/op
BenchmarkIngester_QueryStreamChunks_MatcherOptimization/metric_name_with_regex_matchers_optimization_disabled-10              26332            926871 ns/op    1188941 B/op      13047 allocs/op
BenchmarkIngester_QueryStreamChunks_MatcherOptimization/metric_name_with_regex_matchers_optimization_enabled-10               28640            837144 ns/op    1185907 B/op      13037 allocs/op
BenchmarkIngester_QueryStreamChunks_MatcherOptimization/metric_name_with_not_equal_empty_optimization_disabled-10             24758            976667 ns/op    1276890 B/op      13049 allocs/op
BenchmarkIngester_QueryStreamChunks_MatcherOptimization/metric_name_with_not_equal_empty_optimization_enabled-10              28104            849990 ns/op    1185878 B/op      13037 allocs/op
BenchmarkIngester_QueryStreamChunks_MatcherOptimization/metric_name_with_sparse_label_optimization_disabled-10                53654            462285 ns/op     604060 B/op       6536 allocs/op
BenchmarkIngester_QueryStreamChunks_MatcherOptimization/metric_name_with_sparse_label_optimization_enabled-10                 46366            527558 ns/op     769597 B/op       8531 allocs/op
PASS

@SungJin1212
Copy link
Member

@yeya24
Thanks for sharing

Signed-off-by: Ben Ye <[email protected]>
Signed-off-by: yeya24 <[email protected]>
Signed-off-by: yeya24 <[email protected]>
Signed-off-by: yeya24 <[email protected]>
@yeya24 yeya24 force-pushed the ingester-matcher-optimization branch from 68e6e11 to a564dd2 Compare October 27, 2025 22:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/ingester lgtm This PR has been approved by a maintainer size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants