[improvement](status) Change the return type for block_compression #47566

lzyy2024 · 2025-02-06T16:00:05Z

What problem does this PR solve?

The previous block_compression.cpp didn't make sense for the type of Status returned, so I changed it

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
In the past, a function returned multiple states, perhaps directly returning one state. I made different treatments according to different states

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-02-06T16:00:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zclllyybb · 2025-02-06T16:04:34Z

be/src/util/block_compression.cpp

+            return Status::MemoryLimitExceeded("Fail to do ZLib decompress, error={}",
+                                               zError(zres));
+        } else if (zres == Z_BUF_ERROR) {
+            return Status::InternalError("Fail to do ZLib decompress, error={}", zError(zres));
        }


I think here should be a cover branch of zres != Z_OK. and check other changes for this problem

zclllyybb · 2025-02-06T16:07:01Z

run buildall

doris-robot · 2025-02-06T16:45:13Z

TPC-H: Total hot run time: 31632 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1254a09cd576a2ceb61bb6acf2586baf375a8a67, data reload: false

------ Round 1 ----------------------------------
q1	17611	5212	5034	5034
q2	2042	305	170	170
q3	10427	1225	736	736
q4	10283	996	553	553
q5	8138	2354	2324	2324
q6	188	162	129	129
q7	910	750	622	622
q8	9297	1279	1060	1060
q9	4851	4745	4696	4696
q10	6874	2300	1864	1864
q11	480	270	257	257
q12	342	347	214	214
q13	17766	3772	3061	3061
q14	230	231	204	204
q15	508	474	462	462
q16	613	614	594	594
q17	571	850	331	331
q18	6649	6351	6301	6301
q19	1745	965	552	552
q20	315	327	189	189
q21	2949	2163	1973	1973
q22	370	352	306	306
Total cold run time: 103159 ms
Total hot run time: 31632 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5162	5169	5092	5092
q2	233	323	239	239
q3	2151	2664	2296	2296
q4	1453	1840	1396	1396
q5	4201	4257	4094	4094
q6	210	167	125	125
q7	1840	1814	1726	1726
q8	2583	2586	2581	2581
q9	7273	7221	7125	7125
q10	3005	3198	2830	2830
q11	576	510	493	493
q12	665	796	650	650
q13	3353	3880	3325	3325
q14	280	309	288	288
q15	501	465	479	465
q16	631	674	635	635
q17	1138	1605	1328	1328
q18	7544	7380	7374	7374
q19	791	773	830	773
q20	1972	2016	1878	1878
q21	5402	4976	4766	4766
q22	606	562	517	517
Total cold run time: 51570 ms
Total hot run time: 49996 ms

doris-robot · 2025-02-06T16:56:24Z

TeamCity be ut coverage result:
Function Coverage: 42.08% (11018/26184)
Line Coverage: 32.35% (93013/287565)
Region Coverage: 31.52% (47715/151396)
Branch Coverage: 27.53% (24138/87692)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1254a09cd576a2ceb61bb6acf2586baf375a8a67_1254a09cd576a2ceb61bb6acf2586baf375a8a67/report/index.html

doris-robot · 2025-02-06T16:57:14Z

TPC-DS: Total hot run time: 190503 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1254a09cd576a2ceb61bb6acf2586baf375a8a67, data reload: false

query1	1296	971	933	933
query2	6128	1785	1794	1785
query3	10988	4407	4384	4384
query4	53380	25578	23278	23278
query5	5215	552	483	483
query6	384	191	181	181
query7	5029	541	289	289
query8	330	242	227	227
query9	6371	2480	2483	2480
query10	413	320	254	254
query11	15240	14989	14882	14882
query12	160	108	107	107
query13	1142	540	390	390
query14	10017	6693	6756	6693
query15	207	208	196	196
query16	7018	688	456	456
query17	1096	757	601	601
query18	1521	435	342	342
query19	244	197	161	161
query20	132	121	118	118
query21	206	133	115	115
query22	4793	4660	4537	4537
query23	33951	33366	33572	33366
query24	5901	2437	2395	2395
query25	471	458	400	400
query26	711	301	157	157
query27	1881	533	340	340
query28	2841	2389	2394	2389
query29	548	570	427	427
query30	221	186	156	156
query31	884	874	833	833
query32	97	70	64	64
query33	443	357	301	301
query34	786	904	497	497
query35	808	827	763	763
query36	945	1019	892	892
query37	136	111	75	75
query38	4378	4398	4294	4294
query39	1487	1458	1451	1451
query40	211	124	114	114
query41	54	53	48	48
query42	125	111	104	104
query43	494	515	483	483
query44	1368	796	800	796
query45	185	185	169	169
query46	950	1116	663	663
query47	1882	1889	1850	1850
query48	397	443	311	311
query49	705	514	423	423
query50	789	809	441	441
query51	4268	4240	4241	4240
query52	103	104	92	92
query53	242	272	187	187
query54	489	486	420	420
query55	83	84	80	80
query56	265	277	257	257
query57	1183	1211	1113	1113
query58	243	242	245	242
query59	2700	2915	2769	2769
query60	283	286	254	254
query61	124	116	115	115
query62	761	745	656	656
query63	224	186	178	178
query64	1750	1005	668	668
query65	3240	3125	3111	3111
query66	732	392	288	288
query67	15793	15821	15310	15310
query68	4389	792	530	530
query69	482	304	260	260
query70	1207	1098	1111	1098
query71	402	306	254	254
query72	6327	3748	3816	3748
query73	811	790	343	343
query74	9246	9110	8886	8886
query75	3224	3134	2693	2693
query76	2913	1235	759	759
query77	538	364	285	285
query78	10089	10189	9387	9387
query79	2078	849	580	580
query80	827	549	458	458
query81	532	271	232	232
query82	428	153	127	127
query83	192	179	168	168
query84	283	100	73	73
query85	755	347	305	305
query86	348	310	278	278
query87	4635	4470	4615	4470
query88	2912	2188	2262	2188
query89	422	311	284	284
query90	1663	192	189	189
query91	133	134	104	104
query92	123	63	57	57
query93	1474	1042	592	592
query94	637	418	299	299
query95	340	266	256	256
query96	490	612	267	267
query97	2762	2815	2772	2772
query98	231	214	202	202
query99	1352	1411	1285	1285
Total cold run time: 289687 ms
Total hot run time: 190503 ms

doris-robot · 2025-02-06T17:02:30Z

ClickBench: Total hot run time: 30.27 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1254a09cd576a2ceb61bb6acf2586baf375a8a67, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.43	0.42	0.40
query6	1.17	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.60	0.52	0.52
query10	0.59	0.58	0.57
query11	0.16	0.11	0.10
query12	0.14	0.12	0.11
query13	0.61	0.60	0.59
query14	2.69	2.72	2.74
query15	0.91	0.86	0.86
query16	0.37	0.36	0.37
query17	1.03	1.05	1.03
query18	0.21	0.20	0.19
query19	1.91	1.81	1.98
query20	0.01	0.01	0.01
query21	15.37	0.89	0.55
query22	0.76	1.13	0.61
query23	15.06	1.39	0.62
query24	7.03	1.70	0.64
query25	0.54	0.25	0.10
query26	0.55	0.17	0.14
query27	0.05	0.05	0.04
query28	8.90	0.85	0.43
query29	12.92	3.98	3.27
query30	0.25	0.09	0.06
query31	2.84	0.59	0.38
query32	3.22	0.54	0.46
query33	3.00	3.01	2.99
query34	15.72	5.09	4.48
query35	4.52	4.51	4.54
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.74 s
Total hot run time: 30.27 s

zclllyybb · 2025-02-07T12:37:35Z

be/src/vec/functions/function_compress.cpp

@@ -184,7 +184,8 @@ class FunctionUncompress : public IFunction {
            uncompressed_slice = Slice(col_data.data() + idx, length.value);

            Slice compressed_data(data.data + 4, data.size - 4);
-            auto st = compression_codec->decompress(compressed_data, &uncompressed_slice);
+            auto st = RETURN_IF_ERROR(


no need to modify this

zclllyybb

LGTM

github-actions · 2025-02-07T13:19:36Z

PR approved by anyone and no changes requested.

lzyy2024 · 2025-02-07T13:22:43Z

run buildall

github-actions · 2025-02-07T13:24:19Z

PR approved by at least one committer and no changes requested.

doris-robot · 2025-02-07T13:47:10Z

TPC-H: Total hot run time: 31959 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3cd851ba57144518369395453a3ac1ea90f4b016, data reload: false

------ Round 1 ----------------------------------
q1	17628	5326	5225	5225
q2	2050	306	160	160
q3	10511	1321	711	711
q4	10224	1026	537	537
q5	7552	2360	2374	2360
q6	200	174	130	130
q7	911	755	601	601
q8	9297	1466	1258	1258
q9	5036	4792	4921	4792
q10	6851	2345	1909	1909
q11	483	271	251	251
q12	347	358	214	214
q13	17748	3693	3076	3076
q14	242	240	207	207
q15	522	453	486	453
q16	630	612	583	583
q17	570	900	337	337
q18	6740	6249	6185	6185
q19	1218	977	561	561
q20	319	328	194	194
q21	2878	2189	1909	1909
q22	359	323	306	306
Total cold run time: 102316 ms
Total hot run time: 31959 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5325	5291	5298	5291
q2	248	331	229	229
q3	2177	2671	2340	2340
q4	1390	1823	1336	1336
q5	4218	4131	4161	4131
q6	220	161	126	126
q7	1887	1817	1820	1817
q8	2629	2640	2592	2592
q9	7319	7127	7225	7127
q10	2996	3151	2741	2741
q11	571	501	497	497
q12	727	748	618	618
q13	3594	3914	3248	3248
q14	286	293	267	267
q15	508	481	471	471
q16	631	693	639	639
q17	1159	1640	1338	1338
q18	7660	7330	7348	7330
q19	847	911	1145	911
q20	1944	2045	1842	1842
q21	5326	4994	4860	4860
q22	622	632	564	564
Total cold run time: 52284 ms
Total hot run time: 50315 ms

doris-robot · 2025-02-07T13:58:37Z

TPC-DS: Total hot run time: 185333 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3cd851ba57144518369395453a3ac1ea90f4b016, data reload: false

query1	986	416	384	384
query2	6538	1881	1901	1881
query3	6817	217	213	213
query4	26701	24276	23727	23727
query5	4344	690	519	519
query6	294	197	206	197
query7	4613	501	295	295
query8	290	235	220	220
query9	8565	2508	2495	2495
query10	490	305	258	258
query11	16118	15326	15025	15025
query12	158	106	105	105
query13	1666	508	381	381
query14	9274	6634	6927	6634
query15	216	206	187	187
query16	7162	642	485	485
query17	1210	720	557	557
query18	1971	421	321	321
query19	202	188	158	158
query20	123	124	118	118
query21	212	129	108	108
query22	4305	4446	4295	4295
query23	34003	32906	33022	32906
query24	7702	2377	2378	2377
query25	580	474	465	465
query26	1214	267	151	151
query27	2145	509	343	343
query28	3903	2383	2359	2359
query29	756	561	422	422
query30	236	189	162	162
query31	936	838	824	824
query32	73	67	63	63
query33	567	349	345	345
query34	792	876	503	503
query35	833	842	739	739
query36	969	1001	904	904
query37	118	105	76	76
query38	4238	4169	4157	4157
query39	1470	1391	1443	1391
query40	206	118	104	104
query41	54	56	53	53
query42	126	103	107	103
query43	507	516	485	485
query44	1265	778	775	775
query45	180	167	163	163
query46	853	1068	650	650
query47	1773	1785	1685	1685
query48	371	400	293	293
query49	800	498	421	421
query50	682	721	416	416
query51	4253	4233	4225	4225
query52	111	105	91	91
query53	228	251	185	185
query54	473	487	431	431
query55	82	87	85	85
query56	270	273	250	250
query57	1153	1178	1069	1069
query58	250	240	246	240
query59	2608	2778	2582	2582
query60	280	266	267	266
query61	146	120	119	119
query62	805	733	692	692
query63	223	191	191	191
query64	4354	1038	659	659
query65	3332	3206	3215	3206
query66	1134	412	305	305
query67	15818	15867	15373	15373
query68	5883	793	511	511
query69	484	299	273	273
query70	1208	1123	1129	1123
query71	401	300	272	272
query72	5763	3649	3942	3649
query73	755	783	351	351
query74	9145	9090	8696	8696
query75	3207	3202	2750	2750
query76	3163	1193	727	727
query77	486	391	292	292
query78	10484	10453	9702	9702
query79	1658	820	590	590
query80	677	531	460	460
query81	509	286	258	258
query82	219	155	127	127
query83	174	171	165	165
query84	242	94	73	73
query85	733	339	300	300
query86	373	326	292	292
query87	4597	4618	4411	4411
query88	2833	2199	2161	2161
query89	376	315	289	289
query90	1832	194	196	194
query91	130	133	108	108
query92	67	65	63	63
query93	1197	1013	585	585
query94	641	403	307	307
query95	360	270	262	262
query96	474	553	267	267
query97	2836	2936	2739	2739
query98	221	200	199	199
query99	1315	1415	1270	1270
Total cold run time: 267570 ms
Total hot run time: 185333 ms

doris-robot · 2025-02-07T14:03:54Z

ClickBench: Total hot run time: 30.54 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3cd851ba57144518369395453a3ac1ea90f4b016, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.11
query5	0.42	0.42	0.39
query6	1.15	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.52
query10	0.57	0.59	0.58
query11	0.15	0.11	0.11
query12	0.13	0.11	0.11
query13	0.62	0.59	0.60
query14	2.71	2.72	2.82
query15	0.92	0.85	0.84
query16	0.38	0.38	0.38
query17	1.05	1.03	1.04
query18	0.21	0.20	0.20
query19	1.96	1.80	2.01
query20	0.02	0.01	0.01
query21	15.34	0.88	0.55
query22	0.76	1.27	0.74
query23	14.76	1.38	0.61
query24	7.10	1.85	0.60
query25	0.48	0.34	0.15
query26	0.57	0.16	0.15
query27	0.05	0.05	0.05
query28	9.83	0.86	0.41
query29	12.55	3.91	3.31
query30	0.25	0.09	0.06
query31	2.82	0.58	0.38
query32	3.22	0.54	0.46
query33	2.97	3.02	3.03
query34	15.63	5.09	4.53
query35	4.52	4.53	4.53
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.03
query39	0.03	0.02	0.03
query40	0.18	0.13	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.91 s
Total hot run time: 30.54 s

lzyy2024 changed the title ~~[improvement](status) Change the return type for block_compression (#pr)~~ [improvement](status) Change the return type for block_compression Feb 6, 2025

zclllyybb reviewed Feb 6, 2025

View reviewed changes

lzyy2024 requested a review from zclllyybb February 7, 2025 10:29

lzyy2024 added 3 commits February 7, 2025 20:12

start

820e228

cover all situation

2082463

throw memoryLimit

cb73b4e

lzyy2024 force-pushed the CompressReturn branch from 518ecca to cb73b4e Compare February 7, 2025 12:20

zclllyybb reviewed Feb 7, 2025

View reviewed changes

update

3cd851b

lzyy2024 force-pushed the CompressReturn branch from 3014897 to 3cd851b Compare February 7, 2025 12:52

zclllyybb approved these changes Feb 7, 2025

View reviewed changes

github-actions bot added the reviewed label Feb 7, 2025

BiteTheDDDDt approved these changes Feb 7, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improvement](status) Change the return type for block_compression #47566

[improvement](status) Change the return type for block_compression #47566

lzyy2024 commented Feb 6, 2025 •

edited

Loading

hello-stephen commented Feb 6, 2025

zclllyybb Feb 6, 2025

zclllyybb commented Feb 6, 2025

doris-robot commented Feb 6, 2025

doris-robot commented Feb 6, 2025

doris-robot commented Feb 6, 2025

doris-robot commented Feb 6, 2025

zclllyybb Feb 7, 2025

zclllyybb left a comment

github-actions bot commented Feb 7, 2025

lzyy2024 commented Feb 7, 2025

github-actions bot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

[improvement](status) Change the return type for block_compression #47566

Are you sure you want to change the base?

[improvement](status) Change the return type for block_compression #47566

Conversation

lzyy2024 commented Feb 6, 2025 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

hello-stephen commented Feb 6, 2025

zclllyybb Feb 6, 2025

Choose a reason for hiding this comment

zclllyybb commented Feb 6, 2025

doris-robot commented Feb 6, 2025

doris-robot commented Feb 6, 2025

doris-robot commented Feb 6, 2025

doris-robot commented Feb 6, 2025

zclllyybb Feb 7, 2025

Choose a reason for hiding this comment

zclllyybb left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 7, 2025

lzyy2024 commented Feb 7, 2025

github-actions bot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

doris-robot commented Feb 7, 2025

lzyy2024 commented Feb 6, 2025 •

edited

Loading