Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [InvertedIndex](Variant) support inverted index for array type in variant #47688 #48117

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #47688

…iant (#47688)

This pull request includes several changes to support inverted indexes
for arrays and improve the handling of complex data types in the
codebase. The most important changes include modifications to the
inverted index writer, updates to expression evaluation, and the
addition of new regression tests
@Thearas
Copy link
Contributor

Thearas commented Feb 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 20, 2025
@Thearas
Copy link
Contributor

Thearas commented Feb 20, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40787 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37af31eae3967356109f28a6d987155d195cf759, data reload: false

------ Round 1 ----------------------------------
q1	17590	7345	7255	7255
q2	2071	185	167	167
q3	10531	1058	1168	1058
q4	10501	711	697	697
q5	7771	2801	2834	2801
q6	236	147	143	143
q7	999	604	608	604
q8	9346	1933	1998	1933
q9	6476	6395	6405	6395
q10	7017	2286	2310	2286
q11	463	267	262	262
q12	401	208	210	208
q13	17787	3007	3030	3007
q14	238	215	209	209
q15	567	539	539	539
q16	1060	595	580	580
q17	966	535	614	535
q18	7216	6570	6746	6570
q19	1414	1028	1109	1028
q20	473	208	205	205
q21	4175	3330	3325	3325
q22	1115	1014	980	980
Total cold run time: 108413 ms
Total hot run time: 40787 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7482	7358	7318	7318
q2	344	251	238	238
q3	3051	2995	3004	2995
q4	2087	1897	1920	1897
q5	5992	5927	5864	5864
q6	239	150	154	150
q7	2320	1887	1980	1887
q8	3380	3705	3578	3578
q9	9189	9105	8811	8811
q10	3637	3602	3606	3602
q11	606	523	497	497
q12	818	649	620	620
q13	17658	3152	3128	3128
q14	303	275	266	266
q15	568	525	515	515
q16	708	673	655	655
q17	1832	1631	1600	1600
q18	8232	7719	7652	7652
q19	1729	1610	1544	1544
q20	2091	1859	1897	1859
q21	5587	5188	5343	5188
q22	1119	1011	1015	1011
Total cold run time: 78972 ms
Total hot run time: 60875 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198922 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 37af31eae3967356109f28a6d987155d195cf759, data reload: false

query1	1311	919	924	919
query2	6240	2045	2065	2045
query3	10835	4235	4328	4235
query4	66290	29100	23953	23953
query5	4913	462	447	447
query6	408	178	171	171
query7	5650	319	314	314
query8	293	219	217	217
query9	9396	2739	2719	2719
query10	470	285	262	262
query11	17722	15423	15919	15423
query12	153	108	106	106
query13	1552	457	427	427
query14	9914	7854	7026	7026
query15	211	195	194	194
query16	7274	430	488	430
query17	1111	586	554	554
query18	1846	307	303	303
query19	221	152	155	152
query20	119	108	111	108
query21	210	102	99	99
query22	4717	4471	4553	4471
query23	34258	34220	34101	34101
query24	6102	2965	2924	2924
query25	527	399	409	399
query26	659	162	163	162
query27	1871	352	351	351
query28	4209	2457	2451	2451
query29	695	474	455	455
query30	243	173	170	170
query31	1041	807	834	807
query32	70	60	62	60
query33	434	296	314	296
query34	915	520	519	519
query35	836	757	747	747
query36	1056	990	991	990
query37	131	81	76	76
query38	4144	4125	4130	4125
query39	1550	1497	1495	1495
query40	199	107	113	107
query41	58	59	53	53
query42	119	113	106	106
query43	554	507	503	503
query44	1242	861	855	855
query45	185	177	178	177
query46	1160	730	746	730
query47	2009	1945	1932	1932
query48	474	395	397	395
query49	784	408	411	408
query50	848	441	428	428
query51	7252	7190	7101	7101
query52	105	97	90	90
query53	265	192	191	191
query54	578	472	456	456
query55	78	81	81	81
query56	272	257	275	257
query57	1241	1115	1114	1114
query58	232	220	233	220
query59	3174	3113	2877	2877
query60	277	257	255	255
query61	120	135	105	105
query62	857	732	732	732
query63	214	188	185	185
query64	1405	694	649	649
query65	3262	3230	3203	3203
query66	703	302	310	302
query67	15879	15705	15780	15705
query68	4029	600	596	596
query69	473	264	263	263
query70	1206	1132	1117	1117
query71	373	261	257	257
query72	6380	3959	4005	3959
query73	751	353	353	353
query74	10058	9299	9275	9275
query75	3340	2678	2677	2677
query76	1745	1030	1041	1030
query77	516	274	295	274
query78	10686	9626	9479	9479
query79	1236	603	598	598
query80	862	437	437	437
query81	509	248	246	246
query82	1263	125	121	121
query83	171	168	149	149
query84	280	82	75	75
query85	827	298	286	286
query86	333	300	294	294
query87	4523	4236	4238	4236
query88	3630	2420	2383	2383
query89	428	295	296	295
query90	2050	192	191	191
query91	186	151	151	151
query92	70	48	53	48
query93	1599	550	558	550
query94	848	295	296	295
query95	361	255	251	251
query96	605	293	285	285
query97	3371	3192	3187	3187
query98	226	220	213	213
query99	1681	1432	1431	1431
Total cold run time: 317759 ms
Total hot run time: 198922 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.34 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 37af31eae3967356109f28a6d987155d195cf759, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.50	0.51	0.60
query6	1.15	0.74	0.74
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.49	0.48
query10	0.53	0.53	0.55
query11	0.14	0.10	0.12
query12	0.15	0.11	0.11
query13	0.60	0.59	0.58
query14	2.86	2.83	2.73
query15	0.90	0.83	0.83
query16	0.39	0.37	0.37
query17	1.05	1.06	1.04
query18	0.24	0.23	0.22
query19	1.97	1.89	1.99
query20	0.02	0.01	0.00
query21	15.36	0.59	0.57
query22	2.43	2.60	1.83
query23	16.90	0.85	0.89
query24	3.28	0.31	1.24
query25	0.11	0.30	0.08
query26	0.37	0.13	0.14
query27	0.04	0.04	0.04
query28	10.98	1.10	1.08
query29	12.60	3.36	3.34
query30	0.24	0.06	0.06
query31	2.85	0.38	0.37
query32	3.28	0.46	0.47
query33	2.94	3.00	3.01
query34	16.88	4.49	4.51
query35	4.49	4.53	4.59
query36	0.68	0.48	0.48
query37	0.10	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.01 s
Total hot run time: 32.34 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants