Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug Report: VTGate VStream API Does not Properly Handle Multiple Copy Phase Cycles #17778

Open
mattlord opened this issue Feb 13, 2025 · 0 comments · May be fixed by #17783
Open

Bug Report: VTGate VStream API Does not Properly Handle Multiple Copy Phase Cycles #17778

mattlord opened this issue Feb 13, 2025 · 0 comments · May be fixed by #17783

Comments

@mattlord
Copy link
Contributor

mattlord commented Feb 13, 2025

Overview of the Issue

It is expected that the rowstreamer query used for the copy phase has its execution interrupted at the configured vreplication_copy_phase_duration (default 1h) time. This should not be an error that bubbles up and ends the client stream in the vstreamManager in vtgate. Today, the MySQL error on the tablet does bubble back up to the client and end the stream:

Query execution was interrupted, maximum statement execution time exceeded (errno 3024) (sqlstate HY000)

Reproduction Steps

git checkout main && make build

cd examples/local
alias vtctldclient='command vtctldclient --server=localhost:15999'
# Apply the following diff
diff --git a/examples/common/scripts/vttablet-up.sh b/examples/common/scripts/vttablet-up.sh
index 282cd0553e..0e886d060f 100755
--- a/examples/common/scripts/vttablet-up.sh
+++ b/examples/common/scripts/vttablet-up.sh
@@ -55,6 +55,7 @@ vttablet \
  --pid_file $VTDATAROOT/$tablet_dir/vttablet.pid \
  --heartbeat_on_demand_duration=5s \
  --pprof-http \
+ --vreplication_copy_phase_duration=5s \
  > $VTDATAROOT/$tablet_dir/vttablet.out 2>&1 &

 # Block waiting for the tablet to be listening
diff --git a/examples/local/vstream_client.go b/examples/local/vstream_client.go
index ab00f83871..44041744c7 100644
--- a/examples/local/vstream_client.go
+++ b/examples/local/vstream_client.go
@@ -38,7 +38,7 @@ import (
 */
 func main() {
        ctx := context.Background()
-       streamCustomer := true
+       streamCustomer := false
        var vgtid *binlogdatapb.VGtid
        if streamCustomer {
                vgtid = &binlogdatapb.VGtid{
./101_initial_cluster.sh

# Load data in the customer table
table_file="${VTDATAROOT}/vt_0000000100/data/vt_commerce/customer.ibd"
commerce_primary_uid=$(vtctldclient GetTablets --keyspace commerce --tablet-type primary --shard "0" | awk '{print $1}' | cut -d- -f2 | bc)

# Generate 5MiB of initial data
size=$((5*1024*1024))
while [[ $(stat -f "%z" "${table_file}") -lt ${size} ]]; do
    command mysql -u root --socket "${VTDATAROOT}/vt_0000000${commerce_primary_uid}/mysql.sock" vt_commerce -e "insert into customer (customer_id, email) values (${RANDOM}*${RANDOM}, '${RANDOM}[email protected]')" 2> /dev/null
done

say "Initial data load completed"

# Grow that to at least 2GiB
size=$((2*1024*1024*1024))
i=1
while [[ $(stat -f "%z" "${table_file}") -lt ${size} ]]; do
    command mysql -u root --socket "${VTDATAROOT}/vt_0000000${commerce_primary_uid}/mysql.sock" vt_commerce -e "insert into customer (email) select concat(${i}, email) from customer limit 5000000"
    let i=i+1
done

say "Full data load completed"


go run vstream_client.go

You will see in the final result:

❯ go run vstream_client.go
...
rpc error: code = Unknown desc = target: commerce.0.primary: vttablet: rpc error: code = Unknown desc = Query execution was interrupted, maximum statement execution time exceeded (errno 3024) (sqlstate HY000)

Binary Version

❯ vtgate --version
vtgate version Version: 22.0.0-SNAPSHOT (Git revision 404a55a5c07a5c0d25a3372e38e662d53b1b1d01 branch 'main') built on Thu Feb 13 12:04:41 EST 2025 by [email protected] using go1.23.5 darwin/arm64

Operating System and Environment details

N/A

Log Fragments

@mattlord mattlord self-assigned this Feb 13, 2025
@github-project-automation github-project-automation bot moved this to Backlog in VReplication Feb 13, 2025
@mattlord mattlord moved this from Backlog to Prioritized in VReplication Feb 13, 2025
@mattlord mattlord moved this from Prioritized to In progress in VReplication Feb 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In progress
Development

Successfully merging a pull request may close this issue.

1 participant