feat(query): add nested loop join for new experimental hash join #18961

forsaken628 · 2025-11-13T04:43:36Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Allow new hash join to switch to nested loop join implementation when the size on the build side is less than the new setting nested_loop_join_threshold. Only support inner join for now.

Tests

Unit Test
Logic Test
Benchmark Test
No Test - Explain why

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe):

This change is

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

src/query/service/src/physical_plans/physical_hash_join.rs

src/query/service/src/pipelines/processors/transforms/new_hash_join/transform_hash_join.rs

src/common/exception/src/exception_into.rs

src/query/expression/src/schema.rs

zhang2014 · 2025-11-23T07:13:54Z

src/query/service/src/pipelines/processors/transforms/new_hash_join/memory/basic.rs

            return;
        }

+        if matches!(


Perhaps we can move the branch logic into the nested loop join implementation? I see it only prevents the real finalize build logic. Please let me know if I'm missing any critical information.

github-actions · 2025-11-23T08:24:09Z

Docker Image for PR

tag: pr-18961-658c957-1763886162

note: this image tag is only available for internal use.

github-actions · 2025-11-23T09:10:35Z

ClickBench Report

forsaken628 · 2025-11-24T02:00:49Z

src/query/service/src/physical_plans/physical_hash_join.rs

+                        .map(|data_type| data_type.remove_nullable().is_bitmap())
+                        .unwrap_or_default() =>
+                {
+                    // no function matches signature `eq(Bitmap NULL, Bitmap NULL)


Is it reasonable that eq (bitmap NULL, bitmap NULL) is only available in join? @sundy-li

We can add more signatures for bitmap types.

eq(bitmap, bitmap) alias to bitmap_count( bitmap_xor(a, b) ) = 0,

bitmap_and_count , bitmap_or_count , bitmap_xor_count

github-actions · 2025-11-24T10:40:53Z

Docker Image for PR

tag: pr-18961-34d5d36-1763980755

note: this image tag is only available for internal use.

github-actions · 2025-11-24T11:09:31Z

ClickBench Report

github-actions · 2025-11-26T03:57:18Z

Docker Image for PR

tag: pr-18961-bbf72d7-1764129337

note: this image tag is only available for internal use.

github-actions · 2025-11-26T04:23:44Z

ClickBench Report

load: https://benchmark.databend.com/clickbench/pr/18961/19688748413/load.html

forsaken628 · 2025-11-27T04:14:39Z

benchmark

version: pr-18961-bbf72d7-1764129337
dataset: tpch_sf_100
table supplier rows: 1000000

SETTINGS (
enable_experimental_new_join=1,
nested_loop_join_threshold=0
)
select    s.*,n.n_name
from      supplier s
join      nation n on s.s_nationkey = n.n_nationkey IGNORE_RESULT;

duration: 687ms

SETTINGS (
enable_experimental_new_join=1,
nested_loop_join_threshold=100
)
select    s.*,n.n_name
from      supplier s
join      nation n on s.s_nationkey = n.n_nationkey IGNORE_RESULT;

duration: 834ms

github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Nov 13, 2025

forsaken628 added 2 commits November 17, 2025 14:13

loop join

3f09220

refine

f822514

forsaken628 force-pushed the loopjoin branch from e90e16f to 1b2c400 Compare November 19, 2025 10:33

forsaken628 added 4 commits November 20, 2025 16:02

fix

be7ccb0

x

d44a8e7

clear

1ac9e5a

x

0fb419e

forsaken628 force-pushed the loopjoin branch from 6ed7c25 to 0fb419e Compare November 20, 2025 08:17

forsaken628 changed the title ~~feat(query): nestedloopjoin~~ feat(query): add nested loop join for new experimental hash join Nov 20, 2025

forsaken628 marked this pull request as ready for review November 20, 2025 08:35

forsaken628 requested a review from zhang2014 November 20, 2025 08:36

chatgpt-codex-connector bot reviewed Nov 20, 2025

View reviewed changes

src/query/service/src/physical_plans/physical_hash_join.rs Show resolved Hide resolved

forsaken628 added 3 commits November 21, 2025 10:54

refine

76e5c0d

fix

2d3932f

x

3c64236

zhang2014 reviewed Nov 23, 2025

View reviewed changes

src/query/service/src/pipelines/processors/transforms/new_hash_join/transform_hash_join.rs Outdated Show resolved Hide resolved

zhang2014 reviewed Nov 23, 2025

View reviewed changes

src/common/exception/src/exception_into.rs Outdated Show resolved Hide resolved

zhang2014 reviewed Nov 23, 2025

View reviewed changes

src/query/expression/src/schema.rs Show resolved Hide resolved

zhang2014 reviewed Nov 23, 2025

View reviewed changes

zhang2014 added the ci-benchmark Benchmark: run all test label Nov 23, 2025

fix

e6e9bfb

forsaken628 commented Nov 24, 2025

View reviewed changes

update

321f36d

forsaken628 force-pushed the loopjoin branch from 3277343 to 321f36d Compare November 24, 2025 07:56

forsaken628 added ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits and removed ci-benchmark Benchmark: run all test labels Nov 24, 2025

forsaken628 added 2 commits November 25, 2025 14:50

update

07fcd44

update

166832c

forsaken628 added ci-benchmark-load Benchmark: run data load test and removed ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits labels Nov 26, 2025

feat(query): add nested loop join for new experimental hash join #18961

Are you sure you want to change the base?

feat(query): add nested loop join for new experimental hash join #18961

Conversation

forsaken628 commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Type of change

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhang2014 Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 23, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Nov 23, 2025

ClickBench Report

Uh oh!

forsaken628 Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

sundy-li Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 24, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Nov 24, 2025

ClickBench Report

Uh oh!

github-actions bot commented Nov 26, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Nov 26, 2025

ClickBench Report

Uh oh!

forsaken628 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

forsaken628 commented Nov 13, 2025 •

edited

Loading

sundy-li Nov 25, 2025 •

edited

Loading

forsaken628 commented Nov 27, 2025 •

edited

Loading