Skip to content

Optimize tag index timeseries lookup - #18753

Merged
jt2594838 merged 3 commits into
apache:masterfrom
Caideyipi:codex/optimize-tag-index-query
Sep 30, 2026
Merged

jt2594838 merged 3 commits into
apache:masterfrom
Caideyipi:codex/optimize-tag-index-query

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • use direct map lookup for exact tag-value predicates instead of scanning every distinct value
  • apply the path pattern before sorting matched measurements
  • add a regression test that verifies direct lookup, path filtering, and result ordering

Test

  • mvn test -pl iotdb-core/datanode -Dtest=TagManagerTest -DfailIfNoTests=false -Dcheckstyle.skip=true
  • mvn spotless:apply -pl iotdb-core/datanode

Manual performance UT

TagManagerPerformanceTest is skipped by default. Enable it in a normal Maven build with:

mvn test -pl iotdb-core/datanode -Dtest=TagManagerPerformanceTest -Diotdb.tag.index.perf.enabled=true -DfailIfNoTests=false

Local index-lookup microbenchmark on Windows 11, Intel Core i9-13900H, and Java HotSpot 17.0.15. The legacy side reproduces the previous tag-value scan, sort, then path filter; the optimized side calls the current TagManager lookup. The test asserts identical ordered results. Values are medians of 5 rounds with 30 warmups: 10,000 distinct tag values (1 exact match), 2,000 candidate nodes (100 path matches), and 50,000 exact-lookup or 1,000 path-filter iterations per round.

Scenario Legacy CPU (us/query) Optimized CPU (us/query) CPU speedup Legacy allocation (B/query) Optimized allocation (B/query) Allocation change
Exact value lookup 63.750 1.250 51.00x 760 872 +14.7%
Path filter before sort 1,609.375 828.125 1.94x 337,720 291,688 -13.6%

Local validation: the default-disabled invocation skipped the benchmark as intended; the enabled test passed; TagManagerTest passed all 4 tests; Spotless and Checkstyle passed.

Build note: full DataNode Maven test-compile failed locally in unrelated main-source/generated fill and mode classes. For these measurements, the current TagManager.java and performance test were compiled with javac against cached 2.0.11 artifacts and run through JUnitCore. The numbers cover index lookup, not an end-to-end query.

@jt2594838 jt2594838 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May add a performance comparison

@jt2594838
jt2594838 merged commit 0c9a5ee into apache:master Sep 30, 2026
37 of 39 checks passed
@jt2594838
jt2594838 deleted the codex/optimize-tag-index-query branch September 30, 2026 07:44
jt2594838 pushed a commit that referenced this pull request Sep 30, 2026
* Optimize tag index timeseries lookup

* Add opt-in tag index performance test

(cherry picked from commit 0c9a5ee)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants