Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](single compaction) use cooldown to determine the primary replica #37563

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

csun5285
Copy link
Contributor

@csun5285 csun5285 commented Jul 9, 2024

Proposed changes

Issue Number: close #xxx

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 9, 2024

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40319 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0f5b93c91e9a581cc7d993b087668748b630a1d4, data reload: false

------ Round 1 ----------------------------------
q1	17620	4461	4294	4294
q2	2010	186	187	186
q3	10465	1148	1015	1015
q4	10184	847	815	815
q5	7561	2675	2643	2643
q6	224	137	136	136
q7	964	598	602	598
q8	9227	2055	2085	2055
q9	8795	6509	6471	6471
q10	9018	3757	3700	3700
q11	452	227	236	227
q12	486	231	236	231
q13	17771	2994	2946	2946
q14	259	228	229	228
q15	529	474	486	474
q16	498	399	379	379
q17	959	753	645	645
q18	8016	7542	7464	7464
q19	4992	1371	1348	1348
q20	646	311	317	311
q21	4937	3814	3901	3814
q22	389	339	340	339
Total cold run time: 116002 ms
Total hot run time: 40319 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4413	4304	4233	4233
q2	362	271	273	271
q3	2982	2835	2838	2835
q4	1991	1676	1726	1676
q5	5631	5505	5453	5453
q6	236	131	133	131
q7	2188	1914	1866	1866
q8	3317	3399	3502	3399
q9	8676	8792	8795	8792
q10	4103	3705	3783	3705
q11	593	500	520	500
q12	806	652	671	652
q13	16171	3179	3174	3174
q14	316	262	285	262
q15	530	483	503	483
q16	499	434	431	431
q17	1803	1505	1482	1482
q18	8198	7926	7781	7781
q19	1783	1717	1593	1593
q20	2205	1892	1862	1862
q21	5079	4808	4812	4808
q22	620	565	574	565
Total cold run time: 72502 ms
Total hot run time: 55954 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174378 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0f5b93c91e9a581cc7d993b087668748b630a1d4, data reload: false

query1	918	376	367	367
query2	6445	2459	2423	2423
query3	6652	201	211	201
query4	24595	17384	17119	17119
query5	3759	493	491	491
query6	258	176	177	176
query7	4578	303	285	285
query8	331	300	274	274
query9	8555	2389	2390	2389
query10	440	307	282	282
query11	11777	9947	9962	9947
query12	122	86	88	86
query13	1655	380	384	380
query14	10368	7839	7739	7739
query15	240	186	185	185
query16	7752	310	305	305
query17	1815	585	541	541
query18	1870	284	334	284
query19	199	147	145	145
query20	89	81	78	78
query21	211	136	127	127
query22	4441	4049	4059	4049
query23	33972	33770	33811	33770
query24	10990	2867	2848	2848
query25	632	391	395	391
query26	1158	154	154	154
query27	2504	278	283	278
query28	7260	2122	2142	2122
query29	915	648	656	648
query30	254	152	162	152
query31	953	800	784	784
query32	95	53	53	53
query33	758	311	299	299
query34	971	498	498	498
query35	703	586	566	566
query36	1120	993	958	958
query37	150	89	84	84
query38	2987	2842	2809	2809
query39	907	814	815	814
query40	207	126	121	121
query41	54	51	49	49
query42	119	103	102	102
query43	572	546	546	546
query44	1225	746	735	735
query45	202	170	164	164
query46	1084	735	708	708
query47	1846	1745	1766	1745
query48	373	293	309	293
query49	854	412	410	410
query50	794	399	401	399
query51	6920	6854	6863	6854
query52	96	94	86	86
query53	364	288	297	288
query54	890	445	447	445
query55	73	70	73	70
query56	296	267	263	263
query57	1171	1104	1063	1063
query58	239	268	266	266
query59	3419	3134	3118	3118
query60	301	269	273	269
query61	96	91	91	91
query62	781	628	669	628
query63	312	287	290	287
query64	9433	2192	1625	1625
query65	3194	3153	3121	3121
query66	745	330	340	330
query67	15505	14951	14901	14901
query68	4485	539	538	538
query69	535	426	364	364
query70	1184	1152	1132	1132
query71	405	291	278	278
query72	7115	5310	5490	5310
query73	767	330	320	320
query74	5900	5442	5454	5442
query75	3442	2640	2680	2640
query76	2636	967	878	878
query77	476	308	322	308
query78	9827	9120	8898	8898
query79	2215	517	512	512
query80	2105	477	471	471
query81	604	217	226	217
query82	720	133	139	133
query83	283	176	175	175
query84	278	90	98	90
query85	1465	326	311	311
query86	448	283	330	283
query87	3254	3067	3081	3067
query88	4155	2440	2451	2440
query89	477	386	382	382
query90	1849	193	189	189
query91	137	104	105	104
query92	69	49	53	49
query93	2725	509	515	509
query94	985	215	215	215
query95	413	325	327	325
query96	636	279	275	275
query97	3182	3054	2998	2998
query98	219	199	195	195
query99	1550	1287	1252	1252
Total cold run time: 279105 ms
Total hot run time: 174378 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0f5b93c91e9a581cc7d993b087668748b630a1d4, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.68	0.07	0.08
query5	0.51	0.49	0.48
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.50	0.49
query10	0.54	0.54	0.53
query11	0.17	0.11	0.12
query12	0.14	0.11	0.12
query13	0.60	0.58	0.58
query14	0.75	0.78	0.79
query15	0.86	0.82	0.81
query16	0.38	0.35	0.36
query17	1.00	1.03	0.97
query18	0.24	0.23	0.22
query19	1.79	1.81	1.67
query20	0.01	0.00	0.00
query21	15.40	0.74	0.64
query22	4.86	5.76	2.35
query23	18.28	1.37	1.28
query24	2.09	0.21	0.23
query25	0.15	0.09	0.08
query26	0.29	0.21	0.20
query27	0.44	0.22	0.23
query28	13.32	1.02	0.98
query29	12.60	3.46	3.40
query30	0.26	0.05	0.05
query31	2.88	0.38	0.39
query32	3.27	0.48	0.48
query33	2.88	2.91	2.89
query34	16.96	4.31	4.39
query35	4.46	4.38	4.39
query36	0.64	0.45	0.50
query37	0.19	0.18	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.27 s
Total hot run time: 30.96 s

@csun5285 csun5285 force-pushed the fix_single_compaction branch from 0f5b93c to 4de8803 Compare July 9, 2024 13:56
@csun5285
Copy link
Contributor Author

csun5285 commented Jul 9, 2024

run buildall

Copy link
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40688 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4de8803f90839df7357540dfea7c14218328dd8a, data reload: false

------ Round 1 ----------------------------------
q1	6208	4590	4374	4374
q2	817	196	194	194
q3	1410	1103	1010	1010
q4	1053	832	821	821
q5	6388	2669	2623	2623
q6	227	140	141	140
q7	952	620	712	620
q8	2286	2072	2088	2072
q9	6626	6540	6622	6540
q10	6929	3702	3742	3702
q11	364	237	238	237
q12	385	233	233	233
q13	17953	3018	3036	3018
q14	262	228	222	222
q15	521	489	484	484
q16	464	380	377	377
q17	980	694	648	648
q18	7891	7467	7465	7465
q19	1645	1438	1613	1438
q20	515	317	328	317
q21	4881	3817	3929	3817
q22	400	336	341	336
Total cold run time: 69157 ms
Total hot run time: 40688 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4315	4279	4272	4272
q2	371	264	270	264
q3	3014	2748	2761	2748
q4	1876	1652	1643	1643
q5	5306	5286	5305	5286
q6	222	132	135	132
q7	2166	1742	1745	1742
q8	3234	3368	3327	3327
q9	8551	8800	8833	8800
q10	4047	3784	3801	3784
q11	589	515	523	515
q12	844	643	679	643
q13	17017	3206	3168	3168
q14	302	278	278	278
q15	554	505	484	484
q16	513	459	462	459
q17	1840	1544	1532	1532
q18	8157	7881	7884	7881
q19	1759	1688	1610	1610
q20	2046	1874	1840	1840
q21	5081	4974	5021	4974
q22	643	544	551	544
Total cold run time: 72447 ms
Total hot run time: 55926 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174884 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4de8803f90839df7357540dfea7c14218328dd8a, data reload: false

query1	922	374	366	366
query2	6305	2339	2339	2339
query3	6656	206	214	206
query4	28401	17462	17262	17262
query5	3918	503	488	488
query6	278	169	160	160
query7	4586	297	282	282
query8	326	286	297	286
query9	8733	2392	2365	2365
query10	440	299	276	276
query11	12518	10146	10191	10146
query12	149	93	82	82
query13	1651	386	384	384
query14	9844	7612	7026	7026
query15	246	195	202	195
query16	7825	316	315	315
query17	1808	553	541	541
query18	1968	291	287	287
query19	205	158	157	157
query20	91	82	83	82
query21	214	126	133	126
query22	4211	3994	4144	3994
query23	34154	33615	33700	33615
query24	10622	2884	2896	2884
query25	643	435	411	411
query26	710	158	171	158
query27	2239	288	288	288
query28	5936	2242	2150	2150
query29	891	653	636	636
query30	258	159	162	159
query31	995	786	769	769
query32	97	56	55	55
query33	659	302	316	302
query34	890	524	532	524
query35	702	614	594	594
query36	1145	972	974	972
query37	143	88	94	88
query38	2965	2799	2813	2799
query39	911	836	832	832
query40	214	130	129	129
query41	55	52	53	52
query42	123	108	107	107
query43	585	548	551	548
query44	1128	762	772	762
query45	186	167	176	167
query46	1099	751	759	751
query47	1928	1924	1867	1867
query48	388	312	316	312
query49	933	432	431	431
query50	801	414	419	414
query51	7051	6869	6797	6797
query52	108	92	96	92
query53	356	301	290	290
query54	857	450	454	450
query55	74	76	73	73
query56	282	267	276	267
query57	1122	1048	1065	1048
query58	241	243	248	243
query59	3359	3175	3262	3175
query60	311	283	279	279
query61	99	100	97	97
query62	799	651	654	651
query63	325	300	295	295
query64	9153	2194	1658	1658
query65	3172	3087	3129	3087
query66	746	325	332	325
query67	15454	15035	15033	15033
query68	4614	534	538	534
query69	673	424	349	349
query70	1165	1145	1147	1145
query71	459	290	291	290
query72	8765	5629	5707	5629
query73	762	324	330	324
query74	6016	5513	5435	5435
query75	4309	2681	2695	2681
query76	3636	979	925	925
query77	735	317	322	317
query78	9564	8991	8973	8973
query79	3010	517	521	517
query80	1530	490	502	490
query81	589	225	215	215
query82	1453	141	134	134
query83	371	171	170	170
query84	277	87	92	87
query85	1360	330	312	312
query86	459	315	304	304
query87	3413	3115	3079	3079
query88	3950	2472	2455	2455
query89	472	385	412	385
query90	1863	200	192	192
query91	132	105	104	104
query92	61	52	48	48
query93	2804	509	514	509
query94	1179	216	209	209
query95	420	325	315	315
query96	603	281	274	274
query97	3201	3019	3013	3013
query98	234	198	201	198
query99	1519	1271	1284	1271
Total cold run time: 285802 ms
Total hot run time: 174884 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.66 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4de8803f90839df7357540dfea7c14218328dd8a, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.06	0.05
query4	1.67	0.08	0.08
query5	0.51	0.50	0.49
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.51	0.49
query10	0.56	0.53	0.55
query11	0.15	0.11	0.12
query12	0.16	0.12	0.13
query13	0.61	0.61	0.59
query14	0.75	0.76	0.79
query15	0.85	0.81	0.82
query16	0.37	0.38	0.37
query17	1.02	1.05	1.06
query18	0.23	0.22	0.23
query19	1.92	1.72	1.77
query20	0.01	0.01	0.02
query21	15.39	0.74	0.64
query22	4.38	7.47	1.95
query23	18.33	1.30	1.24
query24	2.01	0.25	0.22
query25	0.17	0.08	0.08
query26	0.31	0.21	0.21
query27	0.45	0.22	0.22
query28	13.28	1.02	0.99
query29	12.64	3.39	3.40
query30	0.25	0.05	0.06
query31	2.87	0.40	0.39
query32	3.26	0.46	0.47
query33	2.91	2.92	2.82
query34	17.08	4.32	4.31
query35	4.51	4.41	4.43
query36	0.66	0.46	0.50
query37	0.18	0.15	0.16
query38	0.16	0.14	0.16
query39	0.04	0.03	0.03
query40	0.14	0.12	0.13
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.12 s
Total hot run time: 30.66 s

@csun5285 csun5285 force-pushed the fix_single_compaction branch from 4de8803 to 255e0e7 Compare July 10, 2024 07:50
@csun5285
Copy link
Contributor Author

run buildall

@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39891 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 193bb2e1baa0f515fc1e6863377c9fa9c9701715, data reload: false

------ Round 1 ----------------------------------
q1	18606	4398	4399	4398
q2	2508	193	191	191
q3	11608	1149	1088	1088
q4	11441	831	712	712
q5	7657	2706	2644	2644
q6	231	144	135	135
q7	960	614	616	614
q8	9221	2090	2049	2049
q9	8972	6496	6537	6496
q10	8962	3697	3723	3697
q11	453	238	236	236
q12	403	230	239	230
q13	17760	2978	3007	2978
q14	276	237	226	226
q15	531	469	494	469
q16	497	385	378	378
q17	976	607	627	607
q18	8034	7470	7406	7406
q19	1678	1507	1442	1442
q20	675	324	314	314
q21	4979	3234	4071	3234
q22	410	347	348	347
Total cold run time: 116838 ms
Total hot run time: 39891 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4301	4266	4247	4247
q2	384	271	271	271
q3	3022	2761	2756	2756
q4	1914	1585	1594	1585
q5	5302	5316	5285	5285
q6	222	131	130	130
q7	2139	1782	1743	1743
q8	3198	3334	3326	3326
q9	8329	8356	8328	8328
q10	3866	3686	3638	3638
q11	571	485	489	485
q12	778	616	598	598
q13	17468	2979	2996	2979
q14	300	267	271	267
q15	525	494	481	481
q16	466	410	421	410
q17	1775	1479	1473	1473
q18	7699	7576	7377	7377
q19	1730	1665	1672	1665
q20	2024	1785	1763	1763
q21	4893	4753	4653	4653
q22	632	519	548	519
Total cold run time: 71538 ms
Total hot run time: 53979 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171018 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 193bb2e1baa0f515fc1e6863377c9fa9c9701715, data reload: false

query1	919	370	368	368
query2	6455	2436	2210	2210
query3	6642	206	218	206
query4	28304	17632	17307	17307
query5	4213	489	481	481
query6	284	199	170	170
query7	4604	299	304	299
query8	322	314	300	300
query9	8498	2395	2371	2371
query10	448	291	274	274
query11	10441	10190	10220	10190
query12	131	85	83	83
query13	1655	403	376	376
query14	9312	7489	7739	7489
query15	234	190	184	184
query16	7821	307	301	301
query17	1829	542	526	526
query18	1955	272	273	272
query19	205	151	150	150
query20	91	90	91	90
query21	208	127	128	127
query22	4418	4136	4029	4029
query23	33735	33018	32969	32969
query24	11934	2889	2811	2811
query25	668	369	365	365
query26	1779	149	156	149
query27	2938	272	273	272
query28	7483	2068	2094	2068
query29	1115	628	614	614
query30	282	148	147	147
query31	1013	725	745	725
query32	94	51	52	51
query33	774	294	302	294
query34	934	493	496	493
query35	689	572	562	562
query36	1087	885	943	885
query37	158	82	78	78
query38	2896	2733	2768	2733
query39	857	797	799	797
query40	278	120	121	120
query41	54	52	53	52
query42	128	97	102	97
query43	601	574	521	521
query44	1170	740	747	740
query45	196	164	158	158
query46	1085	786	764	764
query47	1858	1763	1814	1763
query48	384	304	299	299
query49	1155	410	412	410
query50	789	396	405	396
query51	6910	6811	6807	6807
query52	109	89	93	89
query53	361	294	294	294
query54	950	461	448	448
query55	74	76	74	74
query56	298	266	268	266
query57	1141	1053	1034	1034
query58	261	266	263	263
query59	3307	3315	3081	3081
query60	308	277	277	277
query61	101	105	156	105
query62	836	663	637	637
query63	330	301	284	284
query64	10407	2178	1626	1626
query65	3183	3114	3125	3114
query66	1377	351	342	342
query67	15241	15174	14840	14840
query68	4584	545	543	543
query69	472	338	326	326
query70	1175	1080	1151	1080
query71	389	291	280	280
query72	7138	2871	2632	2632
query73	742	327	324	324
query74	6051	5481	5550	5481
query75	3598	2704	2708	2704
query76	2820	922	965	922
query77	453	306	359	306
query78	9372	9361	9598	9361
query79	3214	525	530	525
query80	1931	483	482	482
query81	572	219	223	219
query82	706	135	133	133
query83	277	173	172	172
query84	269	93	88	88
query85	1467	324	307	307
query86	376	332	313	313
query87	3256	3166	3130	3130
query88	4487	2490	2496	2490
query89	482	382	403	382
query90	1882	196	193	193
query91	130	105	106	105
query92	60	50	52	50
query93	4568	512	505	505
query94	1034	212	261	212
query95	410	316	315	315
query96	622	277	280	277
query97	3140	3024	2997	2997
query98	223	203	201	201
query99	1595	1246	1293	1246
Total cold run time: 287812 ms
Total hot run time: 171018 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.09 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 193bb2e1baa0f515fc1e6863377c9fa9c9701715, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.64	0.08	0.08
query5	0.51	0.48	0.48
query6	1.14	0.73	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.04
query9	0.55	0.50	0.48
query10	0.54	0.54	0.53
query11	0.15	0.12	0.12
query12	0.14	0.12	0.13
query13	0.60	0.59	0.58
query14	0.76	0.77	0.79
query15	0.86	0.79	0.81
query16	0.35	0.38	0.35
query17	1.00	0.94	0.96
query18	0.23	0.22	0.22
query19	1.75	1.72	1.67
query20	0.01	0.01	0.01
query21	15.39	0.75	0.66
query22	4.91	6.16	2.29
query23	18.30	1.40	1.29
query24	2.16	0.22	0.22
query25	0.14	0.09	0.09
query26	0.32	0.20	0.20
query27	0.45	0.23	0.22
query28	13.27	1.03	1.01
query29	12.64	3.39	3.37
query30	0.26	0.06	0.06
query31	2.85	0.40	0.39
query32	3.25	0.49	0.48
query33	2.86	2.91	2.93
query34	17.11	4.34	4.34
query35	4.46	4.48	4.48
query36	0.67	0.50	0.47
query37	0.18	0.15	0.16
query38	0.15	0.14	0.14
query39	0.05	0.04	0.03
query40	0.16	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.44 s
Total hot run time: 31.09 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants