Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

statistics: parallel async merge global stats to improve performance #59656

Open
L-maple opened this issue Feb 20, 2025 · 1 comment · May be fixed by #59661
Open

statistics: parallel async merge global stats to improve performance #59656

L-maple opened this issue Feb 20, 2025 · 1 comment · May be fixed by #59661
Assignees
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@L-maple
Copy link
Contributor

L-maple commented Feb 20, 2025

Enhancement

Background

In the new TiDB version, async merge global stats feature is supported to reduce memory usage and merge stats time.

Async merge global stats is really important for our business clusters, because more than 100+ partitioned tables exist and each partition is partitioned by day range. The largest number of partition now is about 2200+.

But in my sysbench test, merging 4096 partition stats takes about 8 mins and it brings some risks for online.

Optimization

parallel merge fmsketch/cmsketch/histogramAndTopN, so that the performance of merging 4096 partition stats improves about 30%.

@L-maple L-maple added the type/enhancement The issue or PR belongs to an enhancement. label Feb 20, 2025
@L-maple
Copy link
Contributor Author

L-maple commented Feb 20, 2025

/assign @L-maple

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant