Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I create a cluster with 2M users, and use BR to backup / restore it, meet error #59673

Open
tiancaiamao opened this issue Feb 20, 2025 · 1 comment
Labels

Comments

@tiancaiamao
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

I create a cluster with 2M users:

mysql> use mysql;
Database changed

mysql> select count(*) from user;
+----------+
| count(*) |
+----------+
|  2494895 |
+----------+
1 row in set (0.71 sec)

Use br to backup:

tiup br:nightly backup full --storage "local:///tmp/backup"

and restore it:

tiup playground nightly --host 0.0.0.0  --tiflash 0   --db.binpath ./bin/tidb-server 
tiup br:nightly restore full --storage "local:///tmp/backup"

2. What did you expect to see? (Required)

Success

3. What did you see instead (Required)

tiup br:nightly restore full --storage "local:///tmp/backup"
Starting component br: /home/genius/.tiup/components/br/v9.0.0-alpha-nightly/br restore full --storage local:///tmp/backup
Detail BR log in /tmp/br.log.2025-02-20T15.21.56+0800 
Split&Scatter Regions <---------------------------------------------------------------------------------------> 100.00%
Download&Ingest SST <-----------------------------------------------------------------------------------------> 100.00%
Restore Pipeline <--------------------------------------------------------------------------------------------> 100.00%
[2025/02/20 15:22:19.354 +08:00] [INFO] [collector.go:77] ["Full Restore failed summary"] [total-ranges=240] [ranges-succeed=240] [ranges-failed=0] [merge-ranges=798.965µs] [split-regions=125.662394ms] [restore-files=2.746380622s] [restore-pipeline=5.142510093s] [default-CF-files=47] [write-CF-files=193] [split-keys=9]
Error: error during merging temporary tables into system tables, table: global_priv: [BR:Common:ErrUnknown]failed to execute REPLACE INTO mysql.global_priv(host,user,priv) SELECT host,user,priv FROM __tidb_br_temporary_mysql.global_priv;: [kv:8004]Transaction is too large, size: 104857618
Error: run /home/genius/.tiup/components/br/v9.0.0-alpha-nightly/br (wd:/home/genius/.tiup/data/UdKIDAX) failed: exit status 1

It use a single REPLACE INTO for the mysql tables, when there are 2M users, the transaction is too large.

4. What is your TiDB version? (Required)

master 2025-02-20

@tiancaiamao tiancaiamao added type/bug The issue is confirmed as a bug. component/br This issue is related to BR of TiDB. labels Feb 20, 2025
@tiancaiamao
Copy link
Contributor Author

tiancaiamao commented Feb 20, 2025

If you want to reproduce 2M users, set the @@global.tidb_accelerate_user_creation_update variable, it makes create user faster, 2M users takes about 20-30min

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants