Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] (Part3) Support creating materialized views with common aggregate state functions #51510

Merged

Conversation

LiShuMing
Copy link
Contributor

@LiShuMing LiShuMing commented Sep 27, 2024

Why I'm doing:

Since common aggregate state has been supported in aggregate table, we can extend our materialized views' ability to support common aggregate functions.

And also we can use this mv to rewrite query as below:

CREATE TABLE t1 (
    k1 string NOT NULL,
    k2 string,
    k3 DECIMAL(34,0),
    k4 DATE NOT NULL,
    v1 BIGINT DEFAULT "0"
)
DUPLICATE KEY(k1,  k2, k3,  k4)
DISTRIBUTED BY HASH(k4);

-- sync mv
CREATE MATERIALIZED VIEW test_mv1 as 
SELECT k1, k2, avg_union(avg_state(k3 * 4)) as v1 from t1 where k1 != 'a' group by k1, k2;

-- ok
SELECT k1, k2, avg_union(avg_state(k3)) as v1 from t1 group by k1, k2

-- ok
SELECT k1, k2, avg_merge(avg_state(k3)) as v1 from t1 group by k1, k2;

-- ok
SELECT k1, k2, avg(k3) as v1 from t1 group by k1, k2

Besides aggregate table, we can also support common aggregate state functions in duplicated table/asynchronized materialized views, eg:

CREATE TABLE t1 (
    k1 string NOT NULL,
    k2 string,
    k3 DECIMAL(34,0),
    k4 DATE NOT NULL,
    v1 BIGINT DEFAULT "0"
)
DUPLICATE KEY(k1,  k2, k3,  k4)
DISTRIBUTED BY HASH(k4);

-- async mv
CREATE MATERIALIZED VIEW test_mv1
REFRESH MANUAL
as 
SELECT k1, k2, avg_union(avg_state(k3 * 4)) as v1 from t1 where k1 != 'a' group by k1, k2;

-- ok
SELECT k1, k2, avg_union(avg_state(k3)) as v1 from t1 group by k1, k2

-- ok
SELECT k1, k2, avg_merge(avg_state(k3)) as v1 from t1 group by k1, k2;

-- ok
SELECT k1, k2, avg(k3) as v1 from t1 group by k1, k2

What I'm doing:

  • Support creating materialized views with common aggregate state functions
  • Support materialized view rewrite with common aggregate state functions

Fixes #49000

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

@LiShuMing LiShuMing requested a review from a team as a code owner September 27, 2024 09:17
@LiShuMing LiShuMing requested review from a team as code owners October 8, 2024 10:59
@LiShuMing LiShuMing changed the title [Feature] [WIP] Support creating synchroninzed materialied views with common agg state functions [Feature] Support creating synchroninzed materialied views with common agg state functions Oct 8, 2024
@LiShuMing LiShuMing force-pushed the fix/main/support_agg_state_with_sync_mv branch 2 times, most recently from 9318ddd to 4cb00d5 Compare October 9, 2024 12:17
@LiShuMing LiShuMing changed the title [Feature] Support creating synchroninzed materialied views with common agg state functions [Feature] (Part3) Support creating synchroninzed materialied views with common agg state functions Oct 10, 2024
@LiShuMing LiShuMing changed the title [Feature] (Part3) Support creating synchroninzed materialied views with common agg state functions [Feature] (Part3) Support creating materialized views with common aggregate state functions Oct 10, 2024
@LiShuMing LiShuMing force-pushed the fix/main/support_agg_state_with_sync_mv branch 2 times, most recently from 20c9788 to 98c5299 Compare October 10, 2024 07:51
@LiShuMing LiShuMing force-pushed the fix/main/support_agg_state_with_sync_mv branch from 98c5299 to 18723c8 Compare October 10, 2024 12:06
Copy link

sonarcloud bot commented Oct 10, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

Copy link

[Java-Extensions Incremental Coverage Report]

pass : 0 / 0 (0%)

Copy link

[FE Incremental Coverage Report]

pass : 202 / 237 (85.23%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/sql/optimizer/rule/transformation/materialization/equivalent/AggStateRewriteEquivalent.java 65 85 76.47% [59, 66, 70, 73, 76, 79, 97, 127, 131, 134, 139, 143, 191, 192, 199, 200, 207, 208, 209, 210]
🔵 com/starrocks/sql/ast/CreateMaterializedViewStmt.java 39 48 81.25% [461, 463, 466, 552, 553, 560, 569, 576, 580]
🔵 com/starrocks/sql/optimizer/rule/transformation/materialization/AggregateFunctionRewriter.java 27 31 87.10% [115, 119, 156, 164]
🔵 com/starrocks/sql/optimizer/rule/transformation/materialization/AggregatedMaterializedViewRewriter.java 32 34 94.12% [355, 356]
🔵 com/starrocks/catalog/combinator/AggStateUnionCombinator.java 5 5 100.00% []
🔵 com/starrocks/sql/optimizer/rule/transformation/materialization/equivalent/EquivalentShuttleContext.java 4 4 100.00% []
🔵 com/starrocks/catalog/combinator/AggStateUtils.java 4 4 100.00% []
🔵 com/starrocks/sql/ast/MVColumnItem.java 2 2 100.00% []
🔵 com/starrocks/sql/optimizer/rule/transformation/materialization/EquationRewriter.java 10 10 100.00% []
🔵 com/starrocks/catalog/combinator/AggStateMergeCombinator.java 5 5 100.00% []
🔵 com/starrocks/sql/optimizer/rule/transformation/materialization/MvUtils.java 2 2 100.00% []
🔵 com/starrocks/alter/RollupJobV2.java 2 2 100.00% []
🔵 com/starrocks/catalog/combinator/AggStateCombinator.java 5 5 100.00% []

Copy link

[BE Incremental Coverage Report]

pass : 0 / 0 (0%)

@LiShuMing LiShuMing merged commit 7510a4e into StarRocks:main Oct 12, 2024
49 of 50 checks passed
@LiShuMing LiShuMing deleted the fix/main/support_agg_state_with_sync_mv branch October 12, 2024 09:49
LiShuMing added a commit to LiShuMing/starrocks that referenced this pull request Oct 29, 2024
chaoyli pushed a commit that referenced this pull request Oct 29, 2024
…regate state functions (backport #51510) (#52412)

Signed-off-by: shuming.li <[email protected]>
saahilbarai pushed a commit to pinterest/starrocks that referenced this pull request Oct 30, 2024
ZiheLiu pushed a commit to ZiheLiu/starrocks that referenced this pull request Oct 31, 2024
renzhimin7 pushed a commit to renzhimin7/starrocks that referenced this pull request Nov 7, 2024
…regate state functions (StarRocks#51510)

Signed-off-by: shuming.li <[email protected]>
Signed-off-by: zhiminr.ren <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Generic Aggregate Function State
3 participants