[Feature](privilege) support data mask in doris internal auth #38301

qzsee · 2024-07-24T08:02:44Z

Proposed changes

Before, data masking could only be managed by ranger, and ranger was not applicable to users in most cases. doris built-in permission management did not have the management function of data masking, and now it is added.

supported data mask policy

    MASK_REDACT("Replace lowercase with 'x', uppercase with 'X', digits with '0'",
        "regexp_replace(regexp_replace(regexp_replace({col},'([A-Z])', 'X'),'([a-z])','x'),'([0-9])','0')"),
    MASK_SHOW_LAST_4("Show last 4 characters; replace rest with 'X'",
        "LPAD(RIGHT({col}, 4), CHAR_LENGTH({col}), 'X')"),
    MASK_SHOW_FIRST_4("Show first 4 characters; replace rest with 'x'",
        "RPAD(LEFT({col}, 4), CHAR_LENGTH({col}), 'X')"),
    MASK_HASH("Hash the value of a varchar with sha256",
        "hex(sha2({col}, 256))"),
    MASK_NULL("Replace with NULL", "NULL"),
    MASK_DATE_SHOW_YEAR("Date: show only year",
        "date_trunc({col}, 'year')"),
    MASK_DEFAULT("Replace with data type default",
        "");

create database test;

CREATE TABLE `stu` (
  `id` INT NOT NULL,
  `name` VARCHAR(200) NOT NULL,
  `phone` INT NOT NULL,
  `birth` DATETIME NOT NULL
) ENGINE=OLAP
DUPLICATE KEY(`id`, `name`)
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);

add data mask policy

create data mask policy test on internal.test.stu.phone to 'jack' using MASK_DEFAULT;
create data mask policy test1 on internal.test.stu.id to 'jack' using MASK_NULL;
create data mask policy test2 on internal.test.stu.name to 'jack' using MASK_HASH;

show data mask policy

MySQL [test]> show data mask policy;
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+
| PolicyName | CatalogName | DbName | TableName | ColumnName | DataMaskType | DataMaskDef           | User | Role |
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+
| test       | internal    | test   | stu       | phone      | MASK_DEFAULT |                       | jack | NULL |
| test1      | internal    | test   | stu       | id         | MASK_NULL    | NULL                  | jack | NULL |
| test2      | internal    | test   | stu       | name       | MASK_HASH    | hex(sha2({col}, 256)) | jack | NULL |
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+

drop data mask policy

MySQL [test]> drop data mask policy test;
Query OK, 0 rows affected (0.01 sec)

MySQL [test]> show data mask policy;
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+
| PolicyName | CatalogName | DbName | TableName | ColumnName | DataMaskType | DataMaskDef           | User | Role |
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+
| test1      | internal    | test   | stu       | id         | MASK_NULL    | NULL                  | jack | NULL |
| test2      | internal    | test   | stu       | name       | MASK_HASH    | hex(sha2({col}, 256)) | jack | NULL |
+------------+-------------+--------+-----------+------------+--------------+-----------------------+------+------+

explain

MySQL [test]> explain select * from stu;
+------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                              |
+------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                              |
|   OUTPUT EXPRS:                                                              |
|     id[#4]                                                                   |
|     name[#5]                                                                 |
|     phone[#6]                                                                |
|     birth[#7]                                                                |
|   PARTITION: UNPARTITIONED                                                   |
|                                                                              |
|   HAS_COLO_PLAN_NODE: false                                                  |
|                                                                              |
|   VRESULT SINK                                                               |
|      MYSQL_PROTOCAL                                                          |
|                                                                              |
|   1:VEXCHANGE                                                                |
|      offset: 0                                                               |
|      distribute expr lists:                                                  |
|                                                                              |
| PLAN FRAGMENT 1                                                              |
|                                                                              |
|   PARTITION: HASH_PARTITIONED: id[#0]                                        |
|                                                                              |
|   HAS_COLO_PLAN_NODE: false                                                  |
|                                                                              |
|   STREAM DATA SINK                                                           |
|     EXCHANGE ID: 01                                                          |
|     UNPARTITIONED                                                            |
|                                                                              |
|   0:VOlapScanNode(68)                                                        |
|      TABLE: test.stu(stu), PREAGGREGATION: ON                                |
|      partitions=1/1 (stu)                                                    |
|      tablets=1/1, tabletList=28797                                           |
|      cardinality=1, avgRowSize=0.0, numNodes=1                               |
|      pushAggOp=NONE                                                          |
|      final projections: NULL, hex(sha2(name[#1], 256)), phone[#2], birth[#3] |
|      final project output tuple id: 1                                        |
+------------------------------------------------------------------------------+

Issue Number: close #xxx

doris-robot · 2024-07-24T08:02:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zddr

Can you please add some PR descriptions

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCheckPolicy.java

fe/fe-core/src/main/java/org/apache/doris/policy/PolicyMgr.java

qzsee · 2024-08-16T15:03:32Z

run buildall

doris-robot · 2024-08-16T15:30:12Z

TPC-H: Total hot run time: 38388 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ae811e9e64ebce1ec2fcfedaa276ad160c1162f, data reload: false

------ Round 1 ----------------------------------
q1	17867	4345	4317	4317
q2	2054	227	234	227
q3	11638	952	1092	952
q4	10524	716	721	716
q5	7771	2853	2813	2813
q6	272	165	161	161
q7	1032	674	656	656
q8	9590	2110	2116	2110
q9	8642	6547	6505	6505
q10	7094	2228	2209	2209
q11	500	276	275	275
q12	436	263	258	258
q13	17772	3010	2999	2999
q14	300	261	255	255
q15	576	530	542	530
q16	530	419	418	418
q17	996	734	709	709
q18	7511	6757	6753	6753
q19	6652	1016	1061	1016
q20	729	357	360	357
q21	4311	3205	3108	3108
q22	1130	1059	1044	1044
Total cold run time: 117927 ms
Total hot run time: 38388 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4572	4319	4313	4313
q2	438	312	317	312
q3	2884	2654	2753	2654
q4	1997	1727	1729	1727
q5	5666	5717	5634	5634
q6	259	150	154	150
q7	2134	1762	1757	1757
q8	3262	3469	3477	3469
q9	8762	8815	8735	8735
q10	3592	3352	3292	3292
q11	633	542	521	521
q12	855	682	630	630
q13	16953	3255	3193	3193
q14	321	295	290	290
q15	556	512	516	512
q16	519	452	441	441
q17	1849	1549	1534	1534
q18	8398	7803	7759	7759
q19	9714	1744	1514	1514
q20	2132	1918	1894	1894
q21	13599	5481	5494	5481
q22	1207	1080	1093	1080
Total cold run time: 90302 ms
Total hot run time: 56892 ms

qzsee · 2024-08-19T01:30:27Z

run buildall

doris-robot · 2024-08-19T02:11:15Z

TPC-H: Total hot run time: 37996 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b0383da1930218dfb731890284147b74ced8996, data reload: false

------ Round 1 ----------------------------------
q1	17865	4364	4292	4292
q2	2062	204	211	204
q3	11725	965	1196	965
q4	10505	785	709	709
q5	7781	2841	2844	2841
q6	264	161	157	157
q7	1033	651	673	651
q8	9599	2104	2120	2104
q9	8673	6576	6550	6550
q10	7084	2252	2249	2249
q11	518	272	274	272
q12	431	260	261	260
q13	17791	2987	2968	2968
q14	310	258	259	258
q15	560	536	536	536
q16	531	414	418	414
q17	987	669	666	666
q18	7360	6888	6736	6736
q19	5612	1067	1025	1025
q20	727	337	349	337
q21	3878	2775	2787	2775
q22	1106	1027	1039	1027
Total cold run time: 116402 ms
Total hot run time: 37996 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4698	4299	4287	4287
q2	404	318	294	294
q3	2854	2639	2611	2611
q4	1967	1702	1687	1687
q5	5650	5633	5748	5633
q6	233	145	144	144
q7	2188	1742	1750	1742
q8	3273	3512	3513	3512
q9	8778	8747	8714	8714
q10	3541	3382	3335	3335
q11	628	540	521	521
q12	864	664	679	664
q13	17294	3132	3179	3132
q14	318	300	286	286
q15	557	527	513	513
q16	513	442	460	442
q17	1868	1572	1525	1525
q18	8193	7984	7765	7765
q19	8899	1512	1586	1512
q20	2147	1902	1882	1882
q21	12157	5290	5314	5290
q22	1161	1070	1058	1058
Total cold run time: 88185 ms
Total hot run time: 56549 ms

doris-robot · 2024-08-19T02:24:05Z

TPC-DS: Total hot run time: 197140 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b0383da1930218dfb731890284147b74ced8996, data reload: false

query1	1331	911	891	891
query2	6642	2008	1969	1969
query3	10616	3848	3720	3720
query4	57471	23897	23467	23467
query5	5894	725	708	708
query6	548	211	230	211
query7	6387	338	348	338
query8	543	441	452	441
query9	9303	2591	2545	2545
query10	601	347	335	335
query11	17831	14970	15238	14970
query12	208	140	146	140
query13	1701	427	423	423
query14	11980	7456	7673	7456
query15	250	200	193	193
query16	7539	552	555	552
query17	1175	638	610	610
query18	2067	352	364	352
query19	293	172	169	169
query20	145	144	162	144
query21	251	142	142	142
query22	4546	4386	4276	4276
query23	34579	33956	33741	33741
query24	5854	3073	2982	2982
query25	574	441	436	436
query26	742	186	182	182
query27	1897	306	305	305
query28	3922	2221	2206	2206
query29	735	448	454	448
query30	223	202	189	189
query31	1032	870	865	865
query32	106	82	85	82
query33	537	352	353	352
query34	914	527	517	517
query35	890	805	805	805
query36	1126	989	972	972
query37	160	101	117	101
query38	3899	3870	4010	3870
query39	1564	1484	1442	1442
query40	232	153	156	153
query41	139	138	139	138
query42	143	114	115	114
query43	548	520	506	506
query44	1109	785	802	785
query45	229	197	196	196
query46	1133	778	776	776
query47	1915	1809	1818	1809
query48	414	333	331	331
query49	917	597	599	597
query50	872	464	484	464
query51	6886	6717	6804	6717
query52	122	109	110	109
query53	304	224	226	224
query54	612	512	502	502
query55	89	90	89	89
query56	324	309	324	309
query57	1218	1136	1139	1136
query58	311	313	301	301
query59	3112	2804	2948	2804
query60	363	330	322	322
query61	151	144	143	143
query62	791	707	707	707
query63	265	226	228	226
query64	3365	1845	1841	1841
query65	3251	3173	3154	3154
query66	965	672	717	672
query67	15178	14961	15008	14961
query68	8322	592	579	579
query69	707	459	349	349
query70	1182	1159	1157	1157
query71	562	324	320	320
query72	6765	2323	2060	2060
query73	845	359	352	352
query74	9092	8757	8841	8757
query75	3906	2730	2793	2730
query76	4595	1066	1053	1053
query77	851	505	449	449
query78	11120	9753	12778	9753
query79	8840	574	576	574
query80	1663	625	630	625
query81	620	262	259	259
query82	317	158	158	158
query83	358	212	213	212
query84	302	98	95	95
query85	816	355	367	355
query86	369	320	311	311
query87	4520	4117	4265	4117
query88	4390	2455	2452	2452
query89	443	327	323	323
query90	2304	237	235	235
query91	159	164	127	127
query92	86	80	77	77
query93	1077	556	548	548
query94	1101	331	329	329
query95	386	294	297	294
query96	593	292	290	290
query97	3227	3072	3097	3072
query98	247	233	227	227
query99	1569	1268	1305	1268
Total cold run time: 332368 ms
Total hot run time: 197140 ms

doris-robot · 2024-08-19T02:29:40Z

ClickBench: Total hot run time: 31.78 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b0383da1930218dfb731890284147b74ced8996, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.24	0.05	0.06
query4	1.66	0.08	0.08
query5	0.48	0.48	0.49
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.05	0.05
query9	0.56	0.49	0.49
query10	0.54	0.55	0.56
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.63	0.60	0.60
query14	0.78	0.78	0.78
query15	0.87	0.82	0.82
query16	0.36	0.36	0.39
query17	0.98	0.99	0.96
query18	0.23	0.22	0.21
query19	1.91	1.82	1.74
query20	0.01	0.02	0.01
query21	15.40	0.66	0.66
query22	4.78	6.36	2.47
query23	18.34	1.36	1.40
query24	2.10	0.22	0.23
query25	0.15	0.09	0.08
query26	0.32	0.23	0.24
query27	0.47	0.23	0.22
query28	13.30	1.01	1.00
query29	12.64	3.30	3.31
query30	0.39	0.21	0.20
query31	2.82	0.41	0.41
query32	3.23	0.49	0.49
query33	2.96	2.96	2.98
query34	17.16	4.31	4.35
query35	4.40	4.44	4.41
query36	0.66	0.48	0.48
query37	0.22	0.18	0.17
query38	0.17	0.16	0.17
query39	0.06	0.06	0.06
query40	0.18	0.16	0.15
query41	0.12	0.06	0.07
query42	0.08	0.08	0.07
query43	0.07	0.07	0.06
Total cold run time: 110.93 s
Total hot run time: 31.78 s

qzsee · 2024-08-20T01:47:04Z

run buildall

doris-robot · 2024-08-20T02:29:53Z

TPC-H: Total hot run time: 38002 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ce03c77373d16e926520b11f85aa2cf1640b82a, data reload: false

------ Round 1 ----------------------------------
q1	17959	4397	4307	4307
q2	2060	212	211	211
q3	11860	977	1120	977
q4	10540	756	728	728
q5	7786	2876	2806	2806
q6	265	164	161	161
q7	1018	661	642	642
q8	9597	2097	2106	2097
q9	8806	6581	6577	6577
q10	7086	2243	2210	2210
q11	517	281	266	266
q12	427	268	257	257
q13	18836	2988	2969	2969
q14	324	263	245	245
q15	549	529	531	529
q16	516	430	410	410
q17	981	640	748	640
q18	7358	6729	6777	6729
q19	6394	1092	1036	1036
q20	719	343	345	343
q21	3961	3169	2821	2821
q22	1144	1041	1061	1041
Total cold run time: 118703 ms
Total hot run time: 38002 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4514	4276	4282	4276
q2	410	309	300	300
q3	2835	2664	2724	2664
q4	1982	1714	1671	1671
q5	5643	5731	5717	5717
q6	242	156	169	156
q7	2212	1796	1801	1796
q8	3316	3519	3476	3476
q9	8778	8785	8756	8756
q10	3614	3343	3343	3343
q11	628	524	535	524
q12	855	668	653	653
q13	17057	3144	3061	3061
q14	315	305	313	305
q15	587	536	539	536
q16	520	494	470	470
q17	1853	1565	1537	1537
q18	8189	7932	7685	7685
q19	9494	1648	1726	1648
q20	2184	1920	1886	1886
q21	12325	5236	5355	5236
q22	1213	1070	1106	1070
Total cold run time: 88766 ms
Total hot run time: 56766 ms

doris-robot · 2024-08-20T02:42:45Z

TPC-DS: Total hot run time: 196600 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ce03c77373d16e926520b11f85aa2cf1640b82a, data reload: false

query1	1312	909	889	889
query2	6578	2006	1874	1874
query3	10736	4011	3828	3828
query4	59272	24302	23346	23346
query5	6037	732	734	732
query6	555	211	212	211
query7	6452	332	331	331
query8	548	444	436	436
query9	9319	2553	2525	2525
query10	626	354	346	346
query11	18773	15135	15519	15135
query12	207	130	130	130
query13	1711	428	456	428
query14	12504	7610	7291	7291
query15	270	203	198	198
query16	7929	532	514	514
query17	1234	658	625	625
query18	2297	374	348	348
query19	290	197	174	174
query20	160	138	147	138
query21	245	147	164	147
query22	4757	4318	4489	4318
query23	34308	33746	33810	33746
query24	5701	3015	3072	3015
query25	586	441	434	434
query26	732	187	188	187
query27	1840	315	307	307
query28	3978	2204	2126	2126
query29	768	445	459	445
query30	222	179	181	179
query31	1031	837	853	837
query32	121	77	80	77
query33	566	347	343	343
query34	897	501	501	501
query35	863	784	774	774
query36	1115	957	965	957
query37	164	111	102	102
query38	3995	3905	3877	3877
query39	1680	1530	1455	1455
query40	241	160	165	160
query41	140	137	136	136
query42	137	116	119	116
query43	556	507	511	507
query44	1128	800	806	800
query45	227	196	196	196
query46	1133	780	811	780
query47	1908	1863	1878	1863
query48	416	338	331	331
query49	912	582	574	574
query50	863	484	469	469
query51	6845	6778	6805	6778
query52	117	113	113	113
query53	304	228	233	228
query54	614	507	501	501
query55	90	89	91	89
query56	331	325	310	310
query57	1239	1114	1138	1114
query58	303	337	323	323
query59	3076	3008	2850	2850
query60	351	332	335	332
query61	148	143	151	143
query62	795	687	712	687
query63	282	227	228	227
query64	4420	2371	1870	1870
query65	3231	3240	3182	3182
query66	1012	673	669	669
query67	15272	14920	14886	14886
query68	6477	591	586	586
query69	715	428	328	328
query70	1244	1182	1189	1182
query71	541	322	319	319
query72	7045	2305	2075	2075
query73	822	355	366	355
query74	9271	8840	8926	8840
query75	4266	2763	2749	2749
query76	4005	1066	1050	1050
query77	886	443	449	443
query78	9934	8986	9039	8986
query79	6747	566	574	566
query80	1238	631	631	631
query81	605	266	266	266
query82	872	168	167	167
query83	425	217	229	217
query84	321	99	106	99
query85	1048	354	358	354
query86	436	336	303	303
query87	4454	4275	4294	4275
query88	4626	2490	2513	2490
query89	440	324	330	324
query90	2035	232	237	232
query91	155	129	130	129
query92	88	77	76	76
query93	5544	555	549	549
query94	865	286	343	286
query95	395	297	299	297
query96	612	286	284	284
query97	3274	3079	3095	3079
query98	252	230	232	230
query99	1587	1295	1273	1273
Total cold run time: 337695 ms
Total hot run time: 196600 ms

doris-robot · 2024-08-20T02:48:19Z

ClickBench: Total hot run time: 31.64 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4ce03c77373d16e926520b11f85aa2cf1640b82a, data reload: false

query1	0.06	0.05	0.04
query2	0.08	0.05	0.05
query3	0.24	0.06	0.07
query4	1.64	0.09	0.09
query5	0.51	0.49	0.49
query6	1.14	0.73	0.72
query7	0.02	0.02	0.01
query8	0.06	0.05	0.05
query9	0.54	0.49	0.48
query10	0.56	0.53	0.54
query11	0.16	0.12	0.12
query12	0.16	0.13	0.14
query13	0.62	0.60	0.60
query14	0.78	0.79	0.77
query15	0.86	0.82	0.84
query16	0.38	0.39	0.38
query17	1.01	1.00	1.03
query18	0.21	0.21	0.22
query19	1.82	1.78	1.87
query20	0.02	0.01	0.01
query21	15.40	0.66	0.65
query22	4.12	6.97	2.32
query23	18.30	1.47	1.29
query24	2.08	0.24	0.22
query25	0.15	0.08	0.09
query26	0.28	0.19	0.18
query27	0.08	0.09	0.08
query28	13.21	1.04	1.01
query29	12.62	3.35	3.30
query30	0.44	0.26	0.25
query31	2.83	0.41	0.41
query32	3.24	0.50	0.50
query33	2.99	2.94	2.99
query34	16.99	4.37	4.35
query35	4.46	4.49	4.48
query36	0.68	0.49	0.51
query37	0.21	0.18	0.17
query38	0.17	0.17	0.17
query39	0.06	0.06	0.06
query40	0.18	0.15	0.15
query41	0.12	0.07	0.07
query42	0.08	0.07	0.07
query43	0.06	0.07	0.06
Total cold run time: 109.62 s
Total hot run time: 31.64 s

qzsee · 2024-08-20T06:10:03Z

run buildall

fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4

fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CreatePolicyCommand.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropPolicyCommand.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPolicyCommand.java

fe/fe-core/src/main/java/org/apache/doris/nereids/util/RelationUtil.java

fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/InternalAccessController.java

fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnName.java

fe/fe-core/src/main/java/org/apache/doris/policy/DorisDataMaskPolicy.java

qzsee · 2024-08-21T02:32:50Z

run buildall

doris-robot · 2024-08-21T03:12:45Z

TPC-H: Total hot run time: 38256 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ab922a92600decac2ca8b14f7729e015620e877d, data reload: false

------ Round 1 ----------------------------------
q1	17900	4325	4275	4275
q2	2058	217	225	217
q3	11772	1031	1221	1031
q4	10528	725	706	706
q5	7778	2829	2857	2829
q6	266	157	158	157
q7	1011	671	658	658
q8	9578	2105	2124	2105
q9	8754	6555	6548	6548
q10	7091	2245	2238	2238
q11	489	276	271	271
q12	439	266	281	266
q13	19005	2985	3019	2985
q14	298	251	253	251
q15	559	524	524	524
q16	526	418	402	402
q17	990	671	744	671
q18	7307	6796	6704	6704
q19	6316	1121	1102	1102
q20	691	351	334	334
q21	4102	2933	2974	2933
q22	1122	1059	1049	1049
Total cold run time: 118580 ms
Total hot run time: 38256 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4514	4282	4227	4227
q2	398	312	290	290
q3	2854	2655	2701	2655
q4	2006	1701	1712	1701
q5	5621	5699	5661	5661
q6	243	155	165	155
q7	2198	1805	1755	1755
q8	3290	3548	3492	3492
q9	8750	8915	8702	8702
q10	3550	3364	3276	3276
q11	650	535	524	524
q12	862	680	638	638
q13	15879	3210	3198	3198
q14	329	295	291	291
q15	561	520	522	520
q16	490	444	441	441
q17	1858	1517	1552	1517
q18	8676	8169	7948	7948
q19	1760	1564	1499	1499
q20	2173	1927	1903	1903
q21	13046	5422	5361	5361
q22	1173	1091	1069	1069
Total cold run time: 80881 ms
Total hot run time: 56823 ms

qzsee · 2024-08-21T06:12:04Z

run buildall

doris-robot · 2024-08-21T06:52:26Z

TPC-H: Total hot run time: 37850 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97f81fcc30a6d38dfc9786c36ba7a755683492dd, data reload: false

------ Round 1 ----------------------------------
q1	17858	4427	4255	4255
q2	2058	211	216	211
q3	11764	1035	1078	1035
q4	10539	720	711	711
q5	7793	2839	2761	2761
q6	268	157	161	157
q7	1016	652	677	652
q8	9577	2079	2102	2079
q9	8736	6526	6554	6526
q10	7048	2256	2198	2198
q11	502	272	276	272
q12	433	263	263	263
q13	17776	3031	3014	3014
q14	305	262	250	250
q15	550	532	510	510
q16	523	413	410	410
q17	994	676	755	676
q18	7300	6812	6746	6746
q19	6476	1117	1125	1117
q20	667	359	343	343
q21	4299	2868	2660	2660
q22	1105	1014	1004	1004
Total cold run time: 117587 ms
Total hot run time: 37850 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4636	4287	4251	4251
q2	393	300	294	294
q3	2878	2648	2709	2648
q4	1931	1729	1669	1669
q5	5690	5677	5719	5677
q6	243	155	159	155
q7	2196	1794	1793	1793
q8	3292	3534	3459	3459
q9	8797	8708	8762	8708
q10	3595	3349	3323	3323
q11	680	544	530	530
q12	817	691	670	670
q13	17229	3220	3106	3106
q14	317	296	302	296
q15	557	513	512	512
q16	503	463	468	463
q17	1835	1567	1519	1519
q18	8280	7842	7728	7728
q19	8569	1703	1582	1582
q20	2206	1899	1923	1899
q21	13608	5300	5433	5300
q22	1184	1038	1065	1038
Total cold run time: 89436 ms
Total hot run time: 56620 ms

doris-robot · 2024-08-21T07:05:09Z

TPC-DS: Total hot run time: 196962 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97f81fcc30a6d38dfc9786c36ba7a755683492dd, data reload: false

query1	1337	905	872	872
query2	6647	2051	1974	1974
query3	10652	3858	3601	3601
query4	57036	25986	23517	23517
query5	6114	731	723	723
query6	550	219	216	216
query7	6424	347	330	330
query8	546	441	447	441
query9	9356	2522	2508	2508
query10	599	357	339	339
query11	18298	15305	15315	15305
query12	234	135	130	130
query13	1700	461	436	436
query14	11971	7353	7663	7353
query15	279	197	201	197
query16	7146	529	547	529
query17	1148	641	612	612
query18	1897	369	344	344
query19	327	171	172	171
query20	152	137	133	133
query21	250	143	142	142
query22	4404	4432	4425	4425
query23	34637	33569	34045	33569
query24	5895	2974	3002	2974
query25	590	447	440	440
query26	723	193	191	191
query27	1715	309	312	309
query28	3757	2191	2155	2155
query29	722	452	458	452
query30	231	194	196	194
query31	1051	876	867	867
query32	104	77	84	77
query33	524	353	366	353
query34	933	508	546	508
query35	861	791	783	783
query36	1116	948	971	948
query37	158	107	106	106
query38	4004	3890	3828	3828
query39	1497	1492	1464	1464
query40	241	157	154	154
query41	140	137	140	137
query42	134	119	115	115
query43	552	503	518	503
query44	1135	781	802	781
query45	221	196	195	195
query46	1121	793	757	757
query47	1928	1852	1853	1852
query48	404	349	330	330
query49	915	589	584	584
query50	870	476	470	470
query51	7255	7143	7133	7133
query52	121	109	113	109
query53	298	231	231	231
query54	629	519	519	519
query55	91	111	88	88
query56	344	317	319	317
query57	1202	1146	1133	1133
query58	308	310	294	294
query59	3084	2840	2885	2840
query60	360	328	323	323
query61	151	148	147	147
query62	778	701	702	701
query63	260	228	228	228
query64	3340	1859	1876	1859
query65	3269	3201	3218	3201
query66	1060	708	666	666
query67	15396	15160	14978	14978
query68	6282	602	595	595
query69	590	367	326	326
query70	1287	1143	1161	1143
query71	521	326	320	320
query72	6797	2307	2075	2075
query73	819	361	364	361
query74	9343	8888	8751	8751
query75	3657	2728	2713	2713
query76	3319	1118	988	988
query77	766	456	449	449
query78	10015	9094	9064	9064
query79	3890	556	559	556
query80	1512	600	615	600
query81	585	266	260	260
query82	848	166	156	156
query83	372	217	216	216
query84	295	102	102	102
query85	978	359	355	355
query86	454	276	320	276
query87	4483	4219	4222	4219
query88	4706	2510	2516	2510
query89	453	322	354	322
query90	2063	230	225	225
query91	152	126	125	125
query92	93	76	78	76
query93	4906	554	552	552
query94	828	309	325	309
query95	379	293	301	293
query96	638	296	290	290
query97	3243	3054	3072	3054
query98	241	227	227	227
query99	1592	1293	1304	1293
Total cold run time: 327299 ms
Total hot run time: 196962 ms

doris-robot · 2024-08-21T07:10:43Z

ClickBench: Total hot run time: 31.57 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97f81fcc30a6d38dfc9786c36ba7a755683492dd, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.65	0.08	0.08
query5	0.53	0.49	0.49
query6	1.14	0.72	0.74
query7	0.02	0.02	0.02
query8	0.06	0.04	0.05
query9	0.55	0.49	0.50
query10	0.55	0.55	0.55
query11	0.16	0.12	0.12
query12	0.15	0.13	0.13
query13	0.63	0.59	0.58
query14	0.78	0.81	0.78
query15	0.84	0.82	0.84
query16	0.38	0.36	0.38
query17	0.96	1.06	0.98
query18	0.22	0.20	0.21
query19	1.81	1.71	1.76
query20	0.01	0.01	0.02
query21	15.42	0.66	0.65
query22	4.63	6.30	2.47
query23	18.27	1.44	1.32
query24	2.22	0.23	0.22
query25	0.15	0.10	0.09
query26	0.28	0.19	0.18
query27	0.08	0.09	0.08
query28	13.17	1.04	1.02
query29	12.66	3.29	3.28
query30	0.44	0.26	0.24
query31	2.81	0.41	0.40
query32	3.24	0.49	0.48
query33	2.96	2.98	2.93
query34	16.92	4.39	4.36
query35	4.48	4.42	4.42
query36	0.67	0.48	0.50
query37	0.22	0.18	0.18
query38	0.18	0.16	0.16
query39	0.07	0.06	0.06
query40	0.19	0.15	0.16
query41	0.12	0.07	0.06
query42	0.08	0.07	0.07
query43	0.08	0.07	0.06
Total cold run time: 110.14 s
Total hot run time: 31.57 s

lide-reed

LGTM

github-actions · 2024-08-21T09:04:55Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-21T09:04:57Z

PR approved by anyone and no changes requested.

qzsee · 2024-08-22T01:11:04Z

run P0

qzsee · 2024-08-28T14:47:27Z

run buildall

doris-robot · 2024-08-28T15:22:37Z

TPC-H: Total hot run time: 38045 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63e67853eeb83a5749b675fd264a663eb58b5fe6, data reload: false

------ Round 1 ----------------------------------
q1	17598	4433	4263	4263
q2	2011	176	174	174
q3	11873	976	1113	976
q4	10514	757	701	701
q5	7758	2833	2814	2814
q6	229	142	136	136
q7	959	613	599	599
q8	9327	2102	2090	2090
q9	7196	6553	6520	6520
q10	6991	2279	2180	2180
q11	506	239	240	239
q12	393	232	218	218
q13	18742	3071	3023	3023
q14	280	241	233	233
q15	525	497	488	488
q16	576	505	522	505
q17	976	663	684	663
q18	7242	6953	6875	6875
q19	1397	984	1006	984
q20	683	338	341	338
q21	3848	3026	2999	2999
q22	1140	1061	1027	1027
Total cold run time: 110764 ms
Total hot run time: 38045 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4321	4301	4247	4247
q2	389	271	274	271
q3	2864	2701	2684	2684
q4	1930	1653	1625	1625
q5	5611	5672	5721	5672
q6	242	137	141	137
q7	2243	1857	1875	1857
q8	3298	3421	3457	3421
q9	8837	8866	8774	8774
q10	3640	3415	3370	3370
q11	613	499	504	499
q12	850	666	658	658
q13	15837	3125	3255	3125
q14	323	307	278	278
q15	534	492	485	485
q16	625	574	592	574
q17	1847	1549	1538	1538
q18	8135	7964	7921	7921
q19	1746	1561	1583	1561
q20	2187	1905	1913	1905
q21	5887	5593	5394	5394
q22	1137	1062	1071	1062
Total cold run time: 73096 ms
Total hot run time: 57058 ms

doris-robot · 2024-08-28T15:34:53Z

TPC-DS: Total hot run time: 193079 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63e67853eeb83a5749b675fd264a663eb58b5fe6, data reload: false

query1	1276	894	870	870
query2	6427	2026	1943	1943
query3	10765	4113	4040	4040
query4	59877	24680	23394	23394
query5	5495	509	488	488
query6	408	165	168	165
query7	5906	301	301	301
query8	291	206	209	206
query9	9047	2493	2482	2482
query10	482	268	257	257
query11	18047	15050	15484	15050
query12	157	101	100	100
query13	1559	393	372	372
query14	10931	7403	6419	6419
query15	234	181	184	181
query16	7497	521	509	509
query17	1138	576	565	565
query18	1524	307	302	302
query19	275	154	143	143
query20	121	117	116	116
query21	209	103	108	103
query22	4549	4518	4559	4518
query23	34289	33624	33567	33567
query24	5900	2867	2839	2839
query25	561	411	391	391
query26	695	156	152	152
query27	1805	276	275	275
query28	3842	2108	2080	2080
query29	644	404	405	404
query30	238	150	155	150
query31	912	764	746	746
query32	81	55	56	55
query33	485	282	283	282
query34	853	483	470	470
query35	851	753	708	708
query36	1043	977	942	942
query37	150	97	89	89
query38	4065	3864	3940	3864
query39	1479	1389	1372	1372
query40	205	121	122	121
query41	48	47	48	47
query42	122	103	102	102
query43	518	473	475	473
query44	1086	745	754	745
query45	195	163	169	163
query46	1089	772	742	742
query47	1912	1796	1824	1796
query48	366	309	299	299
query49	771	450	443	443
query50	816	433	419	419
query51	7085	7106	7123	7106
query52	98	89	87	87
query53	253	181	178	178
query54	574	464	478	464
query55	81	78	78	78
query56	286	270	273	270
query57	1186	1056	1068	1056
query58	222	239	238	238
query59	3010	2952	2791	2791
query60	318	282	278	278
query61	126	120	125	120
query62	755	667	661	661
query63	213	185	191	185
query64	2979	772	786	772
query65	3226	3145	3148	3145
query66	685	344	353	344
query67	15585	15275	15266	15266
query68	3113	607	613	607
query69	441	284	288	284
query70	1136	1127	1035	1035
query71	374	276	282	276
query72	6479	4155	4273	4155
query73	759	325	331	325
query74	9117	8800	8946	8800
query75	3335	2676	2676	2676
query76	1374	1160	1022	1022
query77	535	326	323	323
query78	9764	9081	9044	9044
query79	1584	537	532	532
query80	844	512	504	504
query81	579	232	228	228
query82	263	150	145	145
query83	227	148	154	148
query84	269	76	73	73
query85	832	315	276	276
query86	362	291	268	268
query87	4249	4275	4204	4204
query88	3280	2337	2325	2325
query89	386	288	279	279
query90	2078	195	190	190
query91	123	101	98	98
query92	63	51	49	49
query93	1767	561	542	542
query94	852	301	303	301
query95	355	266	262	262
query96	599	270	268	268
query97	3215	3090	3071	3071
query98	219	207	206	206
query99	1816	1292	1264	1264
Total cold run time: 308307 ms
Total hot run time: 193079 ms

doris-robot · 2024-08-28T15:40:30Z

ClickBench: Total hot run time: 31.8 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 63e67853eeb83a5749b675fd264a663eb58b5fe6, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.06	0.07
query5	0.52	0.50	0.51
query6	1.13	0.76	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.50
query10	0.54	0.52	0.56
query11	0.15	0.11	0.12
query12	0.15	0.13	0.13
query13	0.61	0.61	0.58
query14	2.03	2.07	2.04
query15	0.84	0.82	0.81
query16	0.37	0.38	0.37
query17	1.00	1.05	1.06
query18	0.21	0.20	0.21
query19	1.86	1.74	1.81
query20	0.01	0.01	0.02
query21	15.39	0.67	0.66
query22	4.69	7.46	1.65
query23	18.30	1.37	1.32
query24	2.16	0.22	0.24
query25	0.15	0.08	0.09
query26	0.27	0.18	0.18
query27	0.07	0.08	0.08
query28	13.18	1.04	1.01
query29	12.60	3.29	3.29
query30	0.25	0.05	0.05
query31	2.90	0.40	0.39
query32	3.27	0.47	0.47
query33	2.97	2.98	2.97
query34	17.01	4.43	4.40
query35	4.46	4.54	4.46
query36	0.66	0.48	0.47
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.13	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 111.16 s
Total hot run time: 31.8 s

morrySnow · 2024-08-29T08:21:29Z

run cloud_p0

github-actions · 2024-08-29T08:21:56Z

PR approved by at least one committer and no changes requested.

lide-reed

LGTM

qzsee · 2024-09-13T01:22:33Z

run buildall

morrySnow · 2024-09-19T10:11:35Z

run buildall

qzsee · 2024-09-24T11:52:05Z

run buildall

doris-robot · 2024-09-24T12:59:21Z

TPC-H: Total hot run time: 40901 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c38ecbfce784ec4c7da85e972b4973d7778e5c8c, data reload: false

------ Round 1 ----------------------------------
q1	17938	7723	7300	7300
q2	2035	292	299	292
q3	12222	1070	1173	1070
q4	10485	772	717	717
q5	7782	2852	2790	2790
q6	241	153	148	148
q7	992	610	615	610
q8	9373	2020	2019	2019
q9	6587	6491	6428	6428
q10	6986	2303	2301	2301
q11	435	252	248	248
q12	412	218	219	218
q13	17787	2985	2972	2972
q14	232	212	212	212
q15	578	523	525	523
q16	672	616	630	616
q17	975	564	608	564
q18	7375	6507	6748	6507
q19	1396	1054	1008	1008
q20	601	291	281	281
q21	4114	3236	3109	3109
q22	1076	968	998	968
Total cold run time: 110294 ms
Total hot run time: 40901 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7297	7286	7232	7232
q2	333	232	229	229
q3	3110	3000	2966	2966
q4	2090	1841	1831	1831
q5	5674	5689	5721	5689
q6	232	147	149	147
q7	2226	1811	1823	1811
q8	3383	3500	3538	3500
q9	8895	9051	8925	8925
q10	3505	3523	3583	3523
q11	602	501	499	499
q12	852	658	595	595
q13	8259	3175	3159	3159
q14	299	276	286	276
q15	577	538	533	533
q16	722	681	666	666
q17	1806	1642	1603	1603
q18	8329	7842	7925	7842
q19	1704	1598	1565	1565
q20	2137	1879	1920	1879
q21	5757	5391	5471	5391
q22	1155	1003	1045	1003
Total cold run time: 68944 ms
Total hot run time: 60864 ms

doris-robot · 2024-09-24T13:11:28Z

TPC-DS: Total hot run time: 192868 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c38ecbfce784ec4c7da85e972b4973d7778e5c8c, data reload: false

query1	963	379	402	379
query2	6370	2128	2060	2060
query3	8688	193	204	193
query4	34050	23635	23521	23521
query5	3453	478	461	461
query6	268	164	165	164
query7	4186	304	311	304
query8	283	216	220	216
query9	9234	2691	2673	2673
query10	471	282	280	280
query11	18148	15288	15302	15288
query12	160	99	98	98
query13	1546	439	430	430
query14	9617	7165	7884	7165
query15	268	176	183	176
query16	8011	465	484	465
query17	1661	627	585	585
query18	2126	321	320	320
query19	388	163	165	163
query20	120	124	118	118
query21	223	113	117	113
query22	5036	4511	4552	4511
query23	35354	34667	34402	34402
query24	11181	2986	2981	2981
query25	623	418	422	418
query26	1193	167	164	164
query27	2219	298	300	298
query28	7555	2496	2447	2447
query29	830	468	440	440
query30	258	160	155	155
query31	1013	800	790	790
query32	102	54	56	54
query33	772	300	305	300
query34	914	517	507	507
query35	894	776	739	739
query36	1109	908	953	908
query37	150	96	93	93
query38	3964	3898	3890	3890
query39	1502	1422	1423	1422
query40	208	99	101	99
query41	52	51	50	50
query42	120	99	100	99
query43	550	492	506	492
query44	1319	808	815	808
query45	200	172	167	167
query46	1151	711	709	709
query47	1961	1846	1885	1846
query48	488	368	387	368
query49	893	411	423	411
query50	831	425	428	425
query51	7113	6987	6901	6901
query52	98	88	87	87
query53	285	186	192	186
query54	1242	471	470	470
query55	82	76	77	76
query56	281	262	273	262
query57	1236	1128	1121	1121
query58	249	242	224	224
query59	3358	3021	2973	2973
query60	284	268	270	268
query61	103	107	105	105
query62	839	685	676	676
query63	220	202	188	188
query64	4066	666	649	649
query65	3253	3204	3204	3204
query66	719	314	337	314
query67	15856	15618	15797	15618
query68	4923	573	562	562
query69	518	299	309	299
query70	1172	1137	1118	1118
query71	347	277	278	277
query72	7546	3954	4011	3954
query73	779	346	350	346
query74	10535	9009	8958	8958
query75	3473	2690	2690	2690
query76	3205	919	866	866
query77	625	300	290	290
query78	10063	9325	9319	9319
query79	1581	592	584	584
query80	2469	450	454	450
query81	569	245	246	245
query82	722	144	140	140
query83	301	135	137	135
query84	284	85	80	80
query85	1582	297	282	282
query86	421	270	275	270
query87	4368	4402	4517	4402
query88	3194	2487	2436	2436
query89	410	283	296	283
query90	1962	189	193	189
query91	184	144	145	144
query92	64	48	50	48
query93	1085	538	537	537
query94	1023	290	312	290
query95	357	271	262	262
query96	624	300	298	298
query97	3282	3163	3150	3150
query98	221	203	193	193
query99	1743	1284	1299	1284
Total cold run time: 301592 ms
Total hot run time: 192868 ms

doris-robot · 2024-09-24T13:16:57Z

ClickBench: Total hot run time: 33.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c38ecbfce784ec4c7da85e972b4973d7778e5c8c, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.52	0.51	0.52
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.54	0.49	0.50
query10	0.55	0.58	0.55
query11	0.14	0.09	0.10
query12	0.14	0.11	0.10
query13	0.62	0.59	0.59
query14	3.10	2.92	2.96
query15	0.88	0.82	0.83
query16	0.41	0.38	0.41
query17	1.00	1.06	0.99
query18	0.19	0.20	0.20
query19	1.89	1.89	1.98
query20	0.01	0.00	0.00
query21	15.35	0.59	0.57
query22	2.38	2.63	2.74
query23	17.35	1.05	0.78
query24	2.64	0.33	0.88
query25	0.11	0.22	0.06
query26	0.43	0.14	0.13
query27	0.05	0.04	0.05
query28	11.74	1.09	1.07
query29	12.55	3.32	3.29
query30	0.24	0.06	0.06
query31	2.89	0.40	0.38
query32	3.26	0.46	0.46
query33	2.97	3.02	3.07
query34	16.75	4.50	4.44
query35	4.47	4.50	4.56
query36	0.66	0.47	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.46 s
Total hot run time: 33.06 s

qzsee · 2024-09-25T04:04:03Z

run cloud_p0

dqz123 · 2025-02-17T09:29:17Z

run buildall

doris-robot · 2025-02-17T09:54:45Z

TPC-H: Total hot run time: 31662 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8f011cd6290513cf910b91ba1fd59477ca98c7db, data reload: false

------ Round 1 ----------------------------------
q1	17616	5332	5104	5104
q2	2041	307	173	173
q3	10391	1342	703	703
q4	10214	1015	553	553
q5	7533	2400	2349	2349
q6	189	167	135	135
q7	888	747	619	619
q8	9309	1267	1130	1130
q9	4964	4573	4816	4573
q10	6820	2311	1881	1881
q11	478	277	262	262
q12	360	353	223	223
q13	17779	3717	3076	3076
q14	222	226	213	213
q15	497	457	463	457
q16	627	610	589	589
q17	588	870	349	349
q18	6743	6202	6272	6202
q19	1082	942	539	539
q20	308	328	197	197
q21	2853	2163	2026	2026
q22	354	319	309	309
Total cold run time: 101856 ms
Total hot run time: 31662 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5164	5120	5196	5120
q2	228	332	232	232
q3	2156	2638	2298	2298
q4	1451	1884	1419	1419
q5	4229	4135	4144	4135
q6	204	163	126	126
q7	1869	1807	1646	1646
q8	2680	2578	2471	2471
q9	7258	7051	7069	7051
q10	2980	3219	2791	2791
q11	568	504	493	493
q12	699	743	617	617
q13	3572	3859	3322	3322
q14	271	307	279	279
q15	506	448	482	448
q16	635	670	632	632
q17	1151	1630	1322	1322
q18	7544	7246	7381	7246
q19	844	890	877	877
q20	1997	2021	1876	1876
q21	5476	5052	4823	4823
q22	617	574	576	574
Total cold run time: 52099 ms
Total hot run time: 49798 ms

doris-robot · 2025-02-17T10:06:12Z

TPC-DS: Total hot run time: 183244 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8f011cd6290513cf910b91ba1fd59477ca98c7db, data reload: false

query1	952	370	385	370
query2	6525	1884	1832	1832
query3	6794	219	216	216
query4	26106	23901	22973	22973
query5	4352	670	518	518
query6	306	245	193	193
query7	4599	517	290	290
query8	308	251	249	249
query9	8625	2494	2471	2471
query10	472	330	258	258
query11	15169	15132	14902	14902
query12	153	113	108	108
query13	1657	529	407	407
query14	9657	6810	6259	6259
query15	228	196	183	183
query16	7787	645	483	483
query17	1213	734	559	559
query18	1999	408	318	318
query19	202	187	161	161
query20	129	117	110	110
query21	205	128	104	104
query22	4312	4306	4580	4306
query23	34378	33400	32953	32953
query24	7781	2383	2369	2369
query25	515	459	386	386
query26	1223	270	151	151
query27	2111	503	334	334
query28	3887	2376	2361	2361
query29	711	535	427	427
query30	233	184	153	153
query31	928	820	767	767
query32	80	63	62	62
query33	560	343	339	339
query34	783	851	509	509
query35	786	835	769	769
query36	965	1019	897	897
query37	128	97	73	73
query38	4240	4201	4118	4118
query39	1600	1445	1381	1381
query40	203	116	106	106
query41	56	55	52	52
query42	135	107	100	100
query43	501	496	480	480
query44	1321	799	776	776
query45	179	172	158	158
query46	853	1044	650	650
query47	1760	1776	1730	1730
query48	377	426	297	297
query49	785	491	417	417
query50	673	750	423	423
query51	4211	4210	4155	4155
query52	120	111	96	96
query53	231	261	182	182
query54	500	482	410	410
query55	81	78	80	78
query56	270	277	248	248
query57	1119	1172	1058	1058
query58	246	242	249	242
query59	2659	2792	2490	2490
query60	290	263	261	261
query61	119	149	120	120
query62	798	727	662	662
query63	236	200	193	193
query64	4257	985	660	660
query65	3282	3143	3155	3143
query66	1053	395	300	300
query67	15898	15572	15406	15406
query68	7997	770	499	499
query69	502	297	263	263
query70	1149	1164	1121	1121
query71	397	305	262	262
query72	5774	3556	3814	3556
query73	743	746	348	348
query74	9252	8942	9016	8942
query75	3185	3174	2685	2685
query76	3287	1173	778	778
query77	492	410	288	288
query78	10043	10008	9355	9355
query79	2022	790	579	579
query80	665	532	450	450
query81	507	270	241	241
query82	202	129	100	100
query83	174	172	150	150
query84	241	89	70	70
query85	750	353	323	323
query86	372	319	295	295
query87	4458	4488	4365	4365
query88	3260	2191	2172	2172
query89	383	310	281	281
query90	2003	189	190	189
query91	207	137	106	106
query92	76	59	54	54
query93	2121	1000	577	577
query94	681	403	292	292
query95	357	268	255	255
query96	490	555	267	267
query97	2740	2858	2732	2732
query98	237	208	205	205
query99	1351	1409	1232	1232
Total cold run time: 270990 ms
Total hot run time: 183244 ms

doris-robot · 2025-02-17T10:11:31Z

ClickBench: Total hot run time: 30.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8f011cd6290513cf910b91ba1fd59477ca98c7db, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.07	0.06
query4	1.61	0.10	0.11
query5	0.41	0.43	0.40
query6	1.16	0.66	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.53
query10	0.58	0.57	0.58
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.62	0.60	0.60
query14	2.68	2.72	2.71
query15	0.92	0.85	0.86
query16	0.38	0.38	0.37
query17	1.04	0.98	1.07
query18	0.22	0.19	0.19
query19	1.92	1.83	2.04
query20	0.01	0.02	0.01
query21	15.37	0.90	0.54
query22	0.75	1.22	0.76
query23	14.85	1.39	0.65
query24	11.55	0.89	0.58
query25	0.38	0.26	0.12
query26	0.90	0.19	0.15
query27	0.05	0.04	0.05
query28	5.98	0.76	0.43
query29	12.61	3.96	3.27
query30	0.25	0.08	0.06
query31	2.84	0.57	0.39
query32	3.22	0.54	0.47
query33	2.94	3.09	3.03
query34	15.70	5.10	4.51
query35	4.56	4.46	4.50
query36	0.66	0.49	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.03
query39	0.02	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.92 s
Total hot run time: 30.4 s

dqz123 · 2025-02-17T12:12:09Z

run p0

morningman self-assigned this Jul 26, 2024

zddr reviewed Jul 30, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCheckPolicy.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/org/apache/doris/policy/PolicyMgr.java Show resolved Hide resolved

qzsee force-pushed the support-data-masking branch from 7a6fd8b to 9ae811e Compare August 16, 2024 15:00

qzsee force-pushed the support-data-masking branch from 4ce03c7 to 397f3a9 Compare August 20, 2024 06:09

morrySnow reviewed Aug 20, 2024

View reviewed changes

qzsee force-pushed the support-data-masking branch from 3df41ac to ab922a9 Compare August 21, 2024 02:26

lide-reed previously approved these changes Aug 21, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 21, 2024

github-actions bot added the reviewed label Aug 21, 2024

morrySnow previously approved these changes Aug 29, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 29, 2024

lide-reed previously approved these changes Sep 5, 2024

View reviewed changes

qzsee dismissed stale reviews from lide-reed and morrySnow via eaaa465 September 13, 2024 01:22

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 13, 2024

qzsee force-pushed the support-data-masking branch from c38ecbf to df30025 Compare November 1, 2024 03:04

morrySnow added the kind/need-document label Nov 4, 2024

qzsee force-pushed the support-data-masking branch from df30025 to c7276bc Compare February 17, 2025 03:02

[Feature](privilege) support data mask in doris internal auth

8f011cd

qzsee force-pushed the support-data-masking branch from 24b7518 to 8f011cd Compare February 17, 2025 09:26

morrySnow added the need more test Add more test label Feb 18, 2025

[Feature](privilege) support data mask in doris internal auth #38301

Are you sure you want to change the base?

[Feature](privilege) support data mask in doris internal auth #38301

Conversation

qzsee commented Jul 24, 2024 • edited Loading

Proposed changes

doris-robot commented Jul 24, 2024

zddr left a comment

Choose a reason for hiding this comment

qzsee commented Aug 16, 2024

doris-robot commented Aug 16, 2024

qzsee commented Aug 19, 2024

doris-robot commented Aug 19, 2024

doris-robot commented Aug 19, 2024

doris-robot commented Aug 19, 2024

qzsee commented Aug 20, 2024

doris-robot commented Aug 20, 2024

doris-robot commented Aug 20, 2024

doris-robot commented Aug 20, 2024

qzsee commented Aug 20, 2024

qzsee commented Aug 21, 2024

doris-robot commented Aug 21, 2024

qzsee commented Aug 21, 2024

doris-robot commented Aug 21, 2024

doris-robot commented Aug 21, 2024

doris-robot commented Aug 21, 2024

lide-reed left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 21, 2024

github-actions bot commented Aug 21, 2024

qzsee commented Aug 22, 2024

qzsee commented Aug 28, 2024

doris-robot commented Aug 28, 2024

doris-robot commented Aug 28, 2024

doris-robot commented Aug 28, 2024

morrySnow commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

lide-reed left a comment

Choose a reason for hiding this comment

qzsee commented Sep 13, 2024

morrySnow commented Sep 19, 2024

qzsee commented Sep 24, 2024

doris-robot commented Sep 24, 2024

doris-robot commented Sep 24, 2024

doris-robot commented Sep 24, 2024

qzsee commented Sep 25, 2024

dqz123 commented Feb 17, 2025

doris-robot commented Feb 17, 2025

doris-robot commented Feb 17, 2025

doris-robot commented Feb 17, 2025

dqz123 commented Feb 17, 2025

qzsee commented Jul 24, 2024 •

edited

Loading