New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP](Load) Supports synchronizing MySQL data through Cdc #35181

Draft

JNSimba wants to merge 14 commits into apache:master from JNSimba:cdc-tvf

Member

JNSimba commented May 22, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

JNSimba added 2 commits

May 22, 2024 11:03


          init

19a3240


          Merge branch 'master' into cdc-tvf

6d76521

# Conflicts:
#	fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java
#	fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/InsertTask.java
#	fe/fe-core/src/main/java/org/apache/doris/job/manager/JobManager.java
#	fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java
#	fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
#	gensrc/proto/internal_service.proto

doris-robot commented May 22, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions bot added the meta-change label

Contributor

github-actions bot commented May 22, 2024

clang-tidy review says "All clean, LGTM! 👍"

JNSimba marked this pull request as draft

May 22, 2024 03:35


          add database cdc

8377a87

Contributor

github-actions bot commented Jun 26, 2024

clang-tidy review says "All clean, LGTM! 👍"

JNSimba added 2 commits

July 1, 2024 14:42


          format code style

26d6f30


          Merge branch 'master' into cdc-tvf

fee7618

# Conflicts:
#	fe/fe-common/src/main/java/org/apache/doris/common/Config.java
#	fe/fe-common/src/main/java/org/apache/doris/common/FeMetaVersion.java
#	fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
#	fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
#	fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java

Contributor

github-actions bot commented Jul 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

JNSimba added 3 commits

July 10, 2024 17:48


          add schema change

21f2e72


          Merge branch 'master' into cdc-tvf

056853b

# Conflicts:
#	fe/fe-common/src/main/java/org/apache/doris/common/FeMetaVersion.java
#	fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java


          reslove confict

6aa4948

Contributor

github-actions bot commented Jul 29, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment

Contributor

github-actions bot commented Jul 29, 2024

clang-tidy review says "All clean, LGTM! 👍"


          update

388e75e

Contributor

github-actions bot commented Jul 29, 2024

clang-tidy review says "All clean, LGTM! 👍"


          add fork cdc process and get split api

4b2fff9

github-actions bot reviewed

View reviewed changes

Contributor

github-actions bot left a comment

clang-tidy made some suggestions

be/src/service/internal_service.cpp

@@ @@ -37,8 +37,16 @@ @@
               #include <gen_cpp/segment_v2.pb.h>
               #include <gen_cpp/types.pb.h>
               #include <google/protobuf/stubs/callback.h>
+              #include <signal.h>

Contributor

github-actions bot Jul 31, 2024

warning: inclusion of deprecated C++ header 'signal.h'; consider using 'csignal' instead [modernize-deprecated-headers]

Suggested change

      
            #include <signal.h>
          
            #include <csignal>

be/src/service/internal_service.cpp Outdated

@@ @@ -943,6 +951,109 @@ @@
                   }
               }
+              static void handle_sigchld(int sig_no) {
+                  for (;;) {
+                      if (waitpid(-1, NULL, WNOHANG) == 0)

Contributor

github-actions bot Jul 31, 2024

warning: use nullptr [modernize-use-nullptr]

Suggested change

      
                    if (waitpid(-1, NULL, WNOHANG) == 0)
          
                    if (waitpid(-1, nullptr, WNOHANG) == 0)

be/src/service/internal_service.cpp Outdated

Comment on lines 956 to 957

		if (waitpid(-1, NULL, WNOHANG) == 0)
		break;

Contributor

github-actions bot Jul 31, 2024

warning: statement should be inside braces [readability-braces-around-statements]

Suggested change

      
                    if (waitpid(-1, NULL, WNOHANG) == 0)
          
                        break;
          
                    if (waitpid(-1, NULL, WNOHANG) == 0) {
          
                        break;
          
            }

Contributor

github-actions bot commented Jul 31, 2024

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/stop_be.sh line 76:
    pgid=$(ps -o pgid= -p ${pid})
                          ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    pgid=$(ps -o pgid= -p "${pid}")

For more information:
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.


          add split and close, disable spring boot static

edfc7e7

github-actions bot added the doing label

Contributor

github-actions bot commented Aug 1, 2024

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/stop_be.sh line 76:
    pgid=$(ps -o pgid= -p ${pid})
                          ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    pgid=$(ps -o pgid= -p "${pid}")

For more information:
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions bot reviewed

View reviewed changes

Contributor

github-actions bot left a comment

clang-tidy made some suggestions

be/src/service/internal_service.cpp Outdated

                   }
               }
+              static void handle_sigchld(int sig_no) {
+                  for (;;) {
+                      if (waitpid(-1, NULL, WNOHANG) == 0){

Contributor

github-actions bot Aug 1, 2024

warning: use nullptr [modernize-use-nullptr]

Suggested change

      
                    if (waitpid(-1, NULL, WNOHANG) == 0){
          
                    if (waitpid(-1, nullptr, WNOHANG) == 0){


          update

41d2aa1

Contributor

github-actions bot commented Aug 2, 2024

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/stop_be.sh line 76:
    pgid=$(eval echo $(ps -o pgid= -p ${pid}))
                     ^----------------------^ SC2046 (warning): Quote this to prevent word splitting.
                                      ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    pgid=$(eval echo $(ps -o pgid= -p "${pid}"))

For more information:
  https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.


          update

1056cd9

Contributor

github-actions bot commented Aug 26, 2024

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/stop_be.sh line 76:
    pgid=$(eval echo $(ps -o pgid= -p ${pid}))
                     ^----------------------^ SC2046 (warning): Quote this to prevent word splitting.
                                      ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    pgid=$(eval echo $(ps -o pgid= -p "${pid}"))

For more information:
  https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.


          reuse binlogclient connection binlogspliterreader

e822e0f

Contributor

github-actions bot commented Aug 30, 2024

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/stop_be.sh line 76:
    pgid=$(eval echo $(ps -o pgid= -p ${pid}))
                     ^----------------------^ SC2046 (warning): Quote this to prevent word splitting.
                                      ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    pgid=$(eval echo $(ps -o pgid= -p "${pid}"))

For more information:
  https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

github-actions[bot] github-actions[bot] left review comments

dataroaring Awaiting requested review from dataroaring dataroaring will be requested when the pull request is marked ready for review dataroaring is a code owner

morningman Awaiting requested review from morningman morningman will be requested when the pull request is marked ready for review morningman is a code owner

yiguolei Awaiting requested review from yiguolei yiguolei will be requested when the pull request is marked ready for review yiguolei is a code owner

xiaokang Awaiting requested review from xiaokang xiaokang will be requested when the pull request is marked ready for review xiaokang is a code owner

CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs will be requested when the pull request is marked ready for review CalvinKirs is a code owner

At least 1 approving review is required to merge this pull request.

Labels

doing meta-change