Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](jdbc catalog) fix gbase catalog time type mapping and scan && limit pushdown #41587

Merged
merged 1 commit into from
Oct 15, 2024

Conversation

zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Oct 9, 2024

followup #41027
We should map time to String and support reading time outside of 24 hours.
and add limit pushdown

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zy-kkk
Copy link
Member Author

zy-kkk commented Oct 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40800 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 325f702d0190b5226c01bcce93d687f4952470b3, data reload: false

------ Round 1 ----------------------------------
q1	17578	7388	8007	7388
q2	2033	280	270	270
q3	12127	1062	1122	1062
q4	10568	733	730	730
q5	7760	2847	2786	2786
q6	248	151	147	147
q7	975	619	612	612
q8	9349	1887	1960	1887
q9	6506	6483	6353	6353
q10	7009	2284	2310	2284
q11	443	251	252	251
q12	404	222	216	216
q13	17783	2958	2982	2958
q14	258	228	210	210
q15	559	537	517	517
q16	636	573	591	573
q17	966	534	517	517
q18	7178	6737	6792	6737
q19	1362	994	1010	994
q20	508	198	195	195
q21	3968	3121	3264	3121
q22	1083	1017	992	992
Total cold run time: 109301 ms
Total hot run time: 40800 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7231	7230	7218	7218
q2	332	234	222	222
q3	2998	2949	2993	2949
q4	2119	1876	1818	1818
q5	5781	5772	5773	5772
q6	232	147	150	147
q7	2214	1873	1833	1833
q8	3363	3572	3421	3421
q9	8964	8844	8895	8844
q10	3595	3573	3494	3494
q11	574	499	495	495
q12	834	664	631	631
q13	8533	3133	3170	3133
q14	303	298	289	289
q15	581	521	523	521
q16	704	656	647	647
q17	1852	1627	1614	1614
q18	8227	7722	7535	7535
q19	1706	1528	1480	1480
q20	2160	1888	1883	1883
q21	5475	5420	5393	5393
q22	1120	1046	1074	1046
Total cold run time: 68898 ms
Total hot run time: 60385 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192061 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 325f702d0190b5226c01bcce93d687f4952470b3, data reload: false

query1	885	398	408	398
query2	6251	2071	1996	1996
query3	8694	200	201	200
query4	34051	23730	23719	23719
query5	3593	476	486	476
query6	272	167	179	167
query7	4212	308	321	308
query8	296	233	227	227
query9	9363	2680	2680	2680
query10	474	287	263	263
query11	17938	15219	15151	15151
query12	149	102	99	99
query13	1587	474	435	435
query14	9929	7484	7659	7484
query15	251	169	171	169
query16	8031	481	532	481
query17	1680	614	591	591
query18	2212	323	328	323
query19	375	155	153	153
query20	121	120	114	114
query21	209	111	105	105
query22	4620	4371	4615	4371
query23	34921	34002	34186	34002
query24	10928	2842	2840	2840
query25	613	413	402	402
query26	1133	162	160	160
query27	2293	303	295	295
query28	7125	2438	2431	2431
query29	835	455	437	437
query30	257	155	153	153
query31	1041	786	808	786
query32	100	58	57	57
query33	780	295	295	295
query34	895	518	510	510
query35	882	744	734	734
query36	1106	932	960	932
query37	150	89	87	87
query38	4074	3986	3948	3948
query39	1492	1424	1409	1409
query40	205	97	96	96
query41	47	45	47	45
query42	111	93	93	93
query43	519	481	482	481
query44	1270	815	822	815
query45	203	165	163	163
query46	1149	738	720	720
query47	1959	1833	1846	1833
query48	424	355	341	341
query49	900	423	409	409
query50	835	413	416	413
query51	6988	6958	6993	6958
query52	101	84	88	84
query53	277	177	182	177
query54	1255	464	464	464
query55	76	73	76	73
query56	284	264	252	252
query57	1236	1147	1144	1144
query58	231	227	233	227
query59	3187	2909	2989	2909
query60	303	275	271	271
query61	102	106	101	101
query62	866	672	652	652
query63	214	182	187	182
query64	3979	640	631	631
query65	3253	3170	3176	3170
query66	761	305	307	305
query67	16018	15518	15668	15518
query68	4619	556	550	550
query69	471	292	298	292
query70	1174	1090	1123	1090
query71	356	269	266	266
query72	7211	3909	3879	3879
query73	775	342	350	342
query74	10300	8939	9007	8939
query75	3434	2698	2665	2665
query76	2951	899	944	899
query77	628	296	306	296
query78	10334	9576	9511	9511
query79	1463	600	598	598
query80	2087	442	458	442
query81	591	239	239	239
query82	738	134	136	134
query83	289	137	138	137
query84	275	76	75	75
query85	1624	303	287	287
query86	443	291	299	291
query87	4668	4302	4356	4302
query88	3648	2432	2352	2352
query89	406	287	281	281
query90	2042	182	179	179
query91	136	110	107	107
query92	61	48	48	48
query93	1075	545	556	545
query94	1147	299	301	299
query95	353	259	253	253
query96	606	277	274	274
query97	3298	3153	3080	3080
query98	231	199	199	199
query99	1559	1295	1306	1295
Total cold run time: 299564 ms
Total hot run time: 192061 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.49 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 325f702d0190b5226c01bcce93d687f4952470b3, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.52	0.51
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.55	0.51	0.50
query10	0.56	0.54	0.53
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.69	2.81	2.78
query15	0.90	0.83	0.83
query16	0.38	0.39	0.38
query17	1.07	1.00	1.03
query18	0.19	0.19	0.19
query19	1.88	1.90	2.01
query20	0.01	0.01	0.02
query21	15.35	0.58	0.59
query22	2.60	2.85	1.88
query23	16.75	1.04	0.83
query24	2.64	0.65	1.13
query25	0.30	0.15	0.10
query26	0.32	0.14	0.14
query27	0.04	0.04	0.04
query28	11.11	1.10	1.08
query29	12.53	3.22	3.26
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.26	0.47	0.46
query33	3.05	3.08	2.96
query34	17.08	4.43	4.49
query35	4.57	4.54	4.48
query36	0.69	0.50	0.48
query37	0.08	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.13	0.12
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.68 s
Total hot run time: 32.49 s

@zy-kkk zy-kkk changed the title [fix](jdbc catalog) fix gbase catalog time type mapping and scan [fix](jdbc catalog) fix gbase catalog time type mapping and scan && limit push Oct 9, 2024
@zy-kkk zy-kkk changed the title [fix](jdbc catalog) fix gbase catalog time type mapping and scan && limit push [fix](jdbc catalog) fix gbase catalog time type mapping and scan && limit pushdown Oct 9, 2024
@zy-kkk
Copy link
Member Author

zy-kkk commented Oct 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40950 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f6b9a01ee32f2023f1e587a955c3fa048e95d62, data reload: false

------ Round 1 ----------------------------------
q1	17589	7610	7251	7251
q2	2034	281	276	276
q3	12129	1109	1201	1109
q4	10560	754	794	754
q5	7752	2900	2804	2804
q6	238	155	148	148
q7	1012	619	626	619
q8	9348	1964	1965	1964
q9	6601	6500	6420	6420
q10	6965	2312	2382	2312
q11	437	253	253	253
q12	422	220	216	216
q13	17770	3014	2995	2995
q14	232	212	218	212
q15	572	526	530	526
q16	634	602	592	592
q17	994	562	585	562
q18	7352	6779	6610	6610
q19	1340	958	964	958
q20	492	207	197	197
q21	3961	3173	3222	3173
q22	1110	999	1011	999
Total cold run time: 109544 ms
Total hot run time: 40950 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7234	7235	7239	7235
q2	327	226	226	226
q3	3063	2968	2956	2956
q4	2081	1857	1878	1857
q5	5843	5778	5768	5768
q6	230	149	150	149
q7	2275	1871	1836	1836
q8	3391	3549	3452	3452
q9	9060	8951	8936	8936
q10	3594	3569	3556	3556
q11	583	488	490	488
q12	812	644	666	644
q13	8582	3202	3139	3139
q14	310	272	275	272
q15	588	510	514	510
q16	693	656	660	656
q17	1866	1644	1566	1566
q18	8237	7816	7671	7671
q19	1703	1436	1442	1436
q20	2098	1871	1926	1871
q21	5674	5331	5476	5331
q22	1136	1073	1023	1023
Total cold run time: 69380 ms
Total hot run time: 60578 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193673 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f6b9a01ee32f2023f1e587a955c3fa048e95d62, data reload: false

query1	935	418	408	408
query2	6270	2100	2060	2060
query3	8688	201	197	197
query4	34028	23718	23592	23592
query5	3532	472	477	472
query6	278	174	172	172
query7	4193	314	315	314
query8	294	228	220	220
query9	9559	2686	2685	2685
query10	475	280	287	280
query11	17807	15205	15172	15172
query12	151	106	100	100
query13	1587	476	460	460
query14	9147	7647	7485	7485
query15	257	174	183	174
query16	8069	488	528	488
query17	1685	643	570	570
query18	2149	309	320	309
query19	382	152	153	152
query20	121	111	111	111
query21	208	101	105	101
query22	4949	4716	4562	4562
query23	34910	34961	35012	34961
query24	11027	2828	2790	2790
query25	544	401	379	379
query26	943	164	162	162
query27	2594	296	299	296
query28	7376	2443	2428	2428
query29	650	418	411	411
query30	292	150	145	145
query31	1059	785	812	785
query32	96	52	51	51
query33	767	289	298	289
query34	940	520	504	504
query35	868	730	724	724
query36	1072	951	980	951
query37	143	83	85	83
query38	4038	3852	3795	3795
query39	1515	1425	1400	1400
query40	253	93	93	93
query41	45	42	41	41
query42	113	98	96	96
query43	528	509	503	503
query44	1269	821	823	821
query45	193	170	163	163
query46	1161	732	724	724
query47	1931	1779	1837	1779
query48	455	356	338	338
query49	904	438	403	403
query50	830	430	428	428
query51	7035	6910	6933	6910
query52	103	91	90	90
query53	272	185	194	185
query54	1092	468	486	468
query55	82	79	80	79
query56	298	270	265	265
query57	1250	1158	1141	1141
query58	257	238	240	238
query59	3340	3253	2985	2985
query60	317	295	272	272
query61	130	131	125	125
query62	826	667	678	667
query63	233	208	192	192
query64	4271	749	726	726
query65	3272	3191	3216	3191
query66	795	316	315	315
query67	15866	15625	15579	15579
query68	3725	589	590	589
query69	513	311	310	310
query70	1208	1145	1078	1078
query71	348	284	271	271
query72	7327	4064	3985	3985
query73	781	361	354	354
query74	9933	8991	9090	8991
query75	3399	2662	2689	2662
query76	3094	1041	884	884
query77	636	313	312	312
query78	10425	9684	9831	9684
query79	2193	607	600	600
query80	2015	469	466	466
query81	587	236	240	236
query82	692	139	135	135
query83	292	140	140	140
query84	292	79	79	79
query85	1269	313	299	299
query86	403	296	303	296
query87	4588	4423	4279	4279
query88	4128	2407	2364	2364
query89	414	297	295	295
query90	1887	184	186	184
query91	145	109	106	106
query92	63	51	49	49
query93	1330	570	551	551
query94	787	290	290	290
query95	356	259	257	257
query96	618	286	279	279
query97	3241	3089	3208	3089
query98	225	201	203	201
query99	1578	1303	1305	1303
Total cold run time: 299211 ms
Total hot run time: 193673 ms

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 9, 2024
Copy link
Contributor

github-actions bot commented Oct 9, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Oct 9, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

ClickBench: Total hot run time: 33.08 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f6b9a01ee32f2023f1e587a955c3fa048e95d62, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.64	0.10	0.10
query5	0.50	0.51	0.52
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.50	0.50
query10	0.57	0.54	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.61	0.63
query14	2.71	2.72	2.72
query15	0.91	0.82	0.84
query16	0.39	0.42	0.38
query17	1.02	1.08	1.06
query18	0.20	0.20	0.19
query19	1.86	1.86	2.05
query20	0.01	0.01	0.01
query21	15.38	0.61	0.61
query22	2.31	2.12	2.05
query23	17.03	0.82	0.79
query24	2.99	1.00	1.07
query25	0.15	0.15	0.10
query26	0.46	0.15	0.14
query27	0.05	0.04	0.04
query28	10.83	1.11	1.07
query29	12.60	3.23	3.23
query30	0.25	0.07	0.06
query31	2.86	0.39	0.38
query32	3.26	0.47	0.46
query33	3.01	3.05	3.06
query34	16.92	4.47	4.43
query35	4.56	4.47	4.49
query36	0.67	0.48	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.6 s
Total hot run time: 33.08 s

@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41066 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04ac8e7ec23781a99c38724fbdfb9a0c6e21a6e9, data reload: false

------ Round 1 ----------------------------------
q1	17572	7451	7299	7299
q2	2022	298	279	279
q3	11984	1090	1216	1090
q4	10564	707	745	707
q5	7754	2869	2787	2787
q6	238	153	148	148
q7	1018	638	606	606
q8	9331	1924	1976	1924
q9	6613	6448	6446	6446
q10	6974	2324	2303	2303
q11	431	254	245	245
q12	413	220	225	220
q13	17767	2969	3022	2969
q14	237	211	218	211
q15	574	521	531	521
q16	644	577	586	577
q17	982	559	553	553
q18	7266	6673	6717	6673
q19	1355	1022	1024	1022
q20	489	198	208	198
q21	4176	3321	3296	3296
q22	1093	992	995	992
Total cold run time: 109497 ms
Total hot run time: 41066 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7208	7218	7278	7218
q2	326	233	227	227
q3	2976	2883	2973	2883
q4	2125	1868	1800	1800
q5	5797	5746	5722	5722
q6	237	147	145	145
q7	2236	1857	1832	1832
q8	3385	3625	3440	3440
q9	8948	8879	8912	8879
q10	3561	3569	3536	3536
q11	579	493	495	493
q12	790	646	637	637
q13	8356	3171	3194	3171
q14	315	266	292	266
q15	587	537	519	519
q16	675	657	637	637
q17	1851	1626	1601	1601
q18	8230	7857	7647	7647
q19	1694	1543	1371	1371
q20	2126	1858	1924	1858
q21	5632	5283	5481	5283
q22	1108	1042	1058	1042
Total cold run time: 68742 ms
Total hot run time: 60207 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191475 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 04ac8e7ec23781a99c38724fbdfb9a0c6e21a6e9, data reload: false

query1	951	410	403	403
query2	6254	2030	2049	2030
query3	8682	193	204	193
query4	34222	23560	23574	23560
query5	3345	486	458	458
query6	264	158	165	158
query7	4195	301	316	301
query8	271	221	222	221
query9	9336	2705	2683	2683
query10	475	314	268	268
query11	17784	15234	15141	15141
query12	155	103	96	96
query13	1585	462	453	453
query14	9776	6898	7647	6898
query15	251	171	173	171
query16	8069	489	428	428
query17	1685	620	591	591
query18	2166	336	337	336
query19	361	156	147	147
query20	118	112	115	112
query21	221	116	108	108
query22	4627	4664	4518	4518
query23	35076	33869	33817	33817
query24	11100	2833	2910	2833
query25	617	414	421	414
query26	1146	165	168	165
query27	2258	299	304	299
query28	7419	2442	2409	2409
query29	820	432	435	432
query30	261	154	152	152
query31	1033	814	799	799
query32	100	56	54	54
query33	756	308	308	308
query34	928	505	523	505
query35	862	736	724	724
query36	1098	947	943	943
query37	152	91	87	87
query38	4015	3875	3887	3875
query39	1463	1443	1405	1405
query40	209	100	97	97
query41	46	43	45	43
query42	128	97	99	97
query43	539	499	475	475
query44	1283	848	821	821
query45	197	165	165	165
query46	1128	725	733	725
query47	1904	1844	1837	1837
query48	438	360	344	344
query49	916	414	409	409
query50	822	427	411	411
query51	7121	6935	6991	6935
query52	106	84	96	84
query53	262	188	188	188
query54	1184	487	488	487
query55	92	79	79	79
query56	303	261	287	261
query57	1240	1132	1174	1132
query58	243	242	257	242
query59	3069	3040	2968	2968
query60	290	269	271	269
query61	105	99	101	99
query62	854	679	668	668
query63	228	188	194	188
query64	4072	634	597	597
query65	3266	3144	3194	3144
query66	776	304	367	304
query67	15961	15620	15669	15620
query68	4443	573	566	566
query69	500	288	301	288
query70	1196	1134	1120	1120
query71	350	273	282	273
query72	7296	3962	3926	3926
query73	768	340	357	340
query74	10218	8892	9046	8892
query75	3448	2700	2661	2661
query76	2894	896	908	896
query77	461	304	308	304
query78	10623	9611	9611	9611
query79	1929	613	605	605
query80	1155	460	465	460
query81	583	241	233	233
query82	647	135	138	135
query83	233	131	137	131
query84	244	79	76	76
query85	1439	289	281	281
query86	415	315	303	303
query87	4428	4241	4408	4241
query88	3283	2429	2365	2365
query89	409	292	289	289
query90	1925	185	186	185
query91	139	102	110	102
query92	67	47	49	47
query93	2027	558	556	556
query94	1030	290	298	290
query95	371	255	257	255
query96	611	278	288	278
query97	3265	3139	3095	3095
query98	216	204	196	196
query99	1549	1291	1303	1291
Total cold run time: 298841 ms
Total hot run time: 191475 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04ac8e7ec23781a99c38724fbdfb9a0c6e21a6e9, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.51	0.51	0.51
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.56	0.50	0.50
query10	0.56	0.56	0.53
query11	0.13	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.59	0.59
query14	2.74	2.71	2.86
query15	0.90	0.84	0.83
query16	0.39	0.38	0.38
query17	1.07	1.02	0.99
query18	0.20	0.20	0.20
query19	1.96	1.80	1.91
query20	0.01	0.01	0.01
query21	15.36	0.61	0.57
query22	2.40	2.03	2.34
query23	17.02	1.09	0.86
query24	3.53	1.86	1.65
query25	0.26	0.10	0.05
query26	0.60	0.13	0.13
query27	0.04	0.03	0.05
query28	9.38	1.08	1.06
query29	12.55	3.26	3.25
query30	0.24	0.06	0.06
query31	2.90	0.39	0.37
query32	3.28	0.47	0.45
query33	3.04	3.07	3.02
query34	16.77	4.47	4.51
query35	4.48	4.52	4.54
query36	0.70	0.48	0.50
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 105.94 s
Total hot run time: 33.51 s

@morningman morningman merged commit ba07103 into apache:master Oct 15, 2024
26 of 27 checks passed
morningman pushed a commit to morningman/doris that referenced this pull request Oct 19, 2024
…imit pushdown (apache#41587)

followup apache#41027
We should map time to String and support reading time outside of 24
hours.
and add limit pushdown
morningman added a commit that referenced this pull request Oct 21, 2024
morningman added a commit to morningman/doris that referenced this pull request Oct 21, 2024
morningman added a commit that referenced this pull request Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants