YQ-4602 supported stream lookup join #25897
Conversation
GrigoriyPA
requested review from
yumkam,
vitstn,
gridnevvvit,
kardymonds and
Copilot
There was a problem hiding this comment.
Pull Request Overview
This PR implements stream lookup join functionality with external data sources in YDB. The feature enables efficient lookup joins where streaming data from one source is joined with lookup data from external sources like S3 or generic connectors.
Key changes include:
- Refactored stream lookup join logic to common DQ optimization modules
- Added support for external data sources in streaming queries
- Implemented caching and validation mechanisms for stream lookup operations
Reviewed Changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| ydb/library/yql/dq/opt/dq_opt_phy.cpp | Moved stream lookup join rewrite logic to common DQ optimization module |
| ydb/library/yql/dq/opt/dq_opt_join.cpp | Moved stream equi-join with lookup rewrite logic to common DQ join module |
| ydb/core/protos/kqp_physical.proto | Added new TKqpPhyCnDqSourceStreamLookup protobuf message for stream lookup connections |
| ydb/core/kqp/query_compiler/kqp_query_compiler.cpp | Added stream lookup connection handling in query compilation |
| ydb/library/yql/providers/generic/connector/libcpp/ut_helpers/connector_client_mock.h | Enhanced mock connector client to support multiple responses and validation options |
| ydb/core/kqp/ut/federated_query/datastreams/datastreams_ut.cpp | Added comprehensive tests for stream lookup join functionality |
Comments suppressed due to low confidence (2)
ydb/core/kqp/ut/federated_query/datastreams/datastreams_ut.cpp:1
- Remove or update the outdated comment '// temp replace with yt source' which doesn't reflect the actual functionality of this function that handles generic lookup sources.
#include <ydb/core/external_sources/external_source.h>
ydb/core/kqp/ut/federated_query/datastreams/datastreams_ut.cpp:1
- The removal of
FillWithDefaults()call from constructor creates inconsistency. Either remove theFillWithDefaults()method entirely or ensure it's called consistently across similar classes in this file.
#include <ydb/core/external_sources/external_source.h>
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
⚪
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ |
|
🟢 |
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
ydb/library/yql/providers/generic/connector/libcpp/ut_helpers/connector_client_mock.h
Show resolved
Hide resolved
ydb/library/yql/providers/generic/connector/libcpp/ut_helpers/connector_client_mock.h
Show resolved
Hide resolved
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
⚪ Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
GrigoriyPA
added
the
rebase-and-check
github-actions
bot
removed
the
rebase-and-check
|
⚪
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
⚪ Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
Тесты на slj в ydb/tests/fq/generic/streaming надо запускать руками (large) ya test --buid relwithdebinfo -ttt ydb/tests/fq/generic/streaming |
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
GrigoriyPA
force-pushed
the
YQ-4602-support-stream-lookup-join
branch
from
0f05be8 to
0eb6597
Compare
|
⚪
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
⚪ Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
GrigoriyPA
added
the
rebase-and-check
github-actions
bot
removed
the
rebase-and-check
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
⚪ Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
|
||
| return Build<TCoEquiJoin>(ctx, node.Pos()).Add(std::move(args)).Done(); | ||
| } | ||
|
|
There was a problem hiding this comment.
сверил, кусок идентичный исходному (с точностью до typeCtx), ok
Changelog entry
Supported stream lookup join with external data sources in YDB
Changelog category
Description for reviewers