br: add table filter for log restore #57394

Tristan1900 · 2024-11-15T01:19:07Z

What problem does this PR solve?

Issue Number: close #57613

Problem Summary:
Need table filter for PiTR

What changed and how does it work?

The following happens if a custom filter is specified during PiTR.

Do a log backup meta scan at first before snapshot restore, and record all the table rename history and table mapping information during the scan, since we don't have access to genGlobalId, we use a temp id as a placeholder.
During snapshot restore, adjust the tables that need to be restored according to the rename history. It needs to add tables that renamed into the filter and removes the tables that are renamed out of the filter range during log backup. And put all the tables that need to be restored into a map/filter and pass down to log restore
During log restore, we use that map/filter from the above and filters out the tables/kvs that don't need to restore/rewrite, and we replace the temp id in the id map with real global ids.
For actual meta kv restore, use id map to filter out the kvs that doesn't need to restore. so for tables not exist in the id map, we can simply ignore it because they are filtered out.
The id map is persisted once built to work with checkpoint, so our filter naturally works with checkpoint as well.

Performance:
we are still scanning log meta kv twice, it's just we move the previously id map building step before snapshot happens.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Add feature to support table level filter for PiTR

ti-chi-bot · 2024-11-15T01:19:12Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

tiprow · 2024-11-15T01:19:25Z

Hi @Tristan1900. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-11-25T01:03:05Z

Codecov Report

Attention: Patch coverage is 80.05698% with 210 lines in your changes missing coverage. Please review.

Project coverage is 74.6340%. Comparing base (ed9b7d5) to head (ebc4f9f).
Report is 1 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57394        +/-   ##
================================================
+ Coverage   73.0629%   74.6340%   +1.5710%     
================================================
  Files          1690       1710        +20     
  Lines        467453     468084       +631     
================================================
+ Hits         341535     349350      +7815     
+ Misses       104943      96970      -7973     
- Partials      20975      21764       +789

Flag	Coverage Δ
integration	`46.0976% <77.3979%> (?)`
unit	`72.1583% <27.7302%> (-0.0829%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`61.6626% <80.0569%> (+15.8682%)`	⬆️

tiprow · 2025-02-06T23:14:24Z

@Tristan1900: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Tristan1900 · 2025-02-06T23:53:56Z

/retest

tiprow · 2025-02-06T23:54:19Z

@Tristan1900: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Leavrth · 2025-02-07T09:01:42Z

br/pkg/utils/filter.go

+
+// Remove removes a table ID from the filter for the given database ID.
+// Returns true if the table was found and removed, false otherwise.
+func (t *PiTRIdTracker) Remove(dbID, physicalId int64) bool {


unused function

Leavrth · 2025-02-07T09:35:18Z

br/pkg/restore/log_client/client.go

@@ -2067,3 +1968,13 @@ func PutRawKvWithRetry(ctx context.Context, client *rawkv.RawKVBatchClient, key,
 	}
 	return nil
 }
+
+// DropTable drops a table with the given database and table name
+func (rc *LogClient) DropTable(ctx context.Context, dbName, tableName string) error {


unused function

Tristan1900 · 2025-02-12T03:52:12Z

/hold
need rebase

ti-chi-bot · 2025-02-12T08:22:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Leavrth, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Leavrth,YuJuncen]
~~br/OWNERS~~ [Leavrth,YuJuncen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-02-12T08:22:43Z

[LGTM Timeline notifier]

Timeline:

2025-02-07 10:02:14.315723209 +0000 UTC m=+5176.711945271: ☑️ agreed by Leavrth.
2025-02-12 08:22:42.428621845 +0000 UTC m=+431204.824843907: ☑️ agreed by YuJuncen.

Signed-off-by: Wenqi Mou <[email protected]>

Tristan1900 · 2025-02-12T23:04:57Z

/unhold

Tristan1900 force-pushed the table-filter branch from 7c04493 to 8ef978b Compare November 22, 2024 01:21

ti-chi-bot bot removed do-not-merge/needs-tests-checked do-not-merge/needs-linked-issue labels Nov 22, 2024

Tristan1900 force-pushed the table-filter branch from 8ef978b to 02a2318 Compare November 25, 2024 00:04

Tristan1900 marked this pull request as ready for review November 25, 2024 00:06

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 25, 2024

Tristan1900 force-pushed the table-filter branch from 02a2318 to c29b578 Compare November 25, 2024 00:18

Tristan1900 force-pushed the table-filter branch 2 times, most recently from f624281 to 6ebc531 Compare November 25, 2024 15:53

Tristan1900 force-pushed the table-filter branch 6 times, most recently from 852dcff to b5132a9 Compare December 23, 2024 18:56

Tristan1900 force-pushed the table-filter branch 7 times, most recently from d2b68b8 to d72415c Compare December 30, 2024 23:30

Leavrth approved these changes Feb 7, 2025

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Feb 7, 2025

Tristan1900 force-pushed the table-filter branch from 9f276bf to fc0e5a5 Compare February 7, 2025 20:59

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 12, 2025

YuJuncen approved these changes Feb 12, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Feb 12, 2025

Tristan1900 added 7 commits February 12, 2025 16:08

rebase on master

d53996a

Signed-off-by: Wenqi Mou <[email protected]>

draft

32ae97d

Signed-off-by: Wenqi Mou <[email protected]>

remove exchange partition as not able to support

cea7b6a

Signed-off-by: Wenqi Mou <[email protected]>

fix go mod

2291b5f

Signed-off-by: Wenqi Mou <[email protected]>

fix tests

3313c2e

Signed-off-by: Wenqi Mou <[email protected]>

fix comments

ac7e8bb

Signed-off-by: Wenqi Mou <[email protected]>

rebase

ebc4f9f

Signed-off-by: Wenqi Mou <[email protected]>

Tristan1900 force-pushed the table-filter branch from fc0e5a5 to ebc4f9f Compare February 12, 2025 21:25

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Feb 12, 2025

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 12, 2025

ti-chi-bot bot merged commit b7e9769 into pingcap:master Feb 12, 2025
28 checks passed

BornChanger mentioned this pull request Feb 12, 2025

br: add table filter for Pitr restore #57613

Closed

7 tasks

This was referenced Feb 21, 2025

br: fix go leak when nothing to restore #58626

Closed

br: goleak detected leaked go routine in int test #58624

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

br: add table filter for log restore #57394

br: add table filter for log restore #57394

Tristan1900 commented Nov 15, 2024 •

edited

Loading

ti-chi-bot bot commented Nov 15, 2024

tiprow bot commented Nov 15, 2024

codecov bot commented Nov 25, 2024 •

edited

Loading

tiprow bot commented Feb 6, 2025

Tristan1900 commented Feb 6, 2025

tiprow bot commented Feb 6, 2025

Leavrth Feb 7, 2025

Leavrth Feb 7, 2025

Tristan1900 commented Feb 12, 2025

ti-chi-bot bot commented Feb 12, 2025

ti-chi-bot bot commented Feb 12, 2025

Tristan1900 commented Feb 12, 2025

br: add table filter for log restore #57394

br: add table filter for log restore #57394

Conversation

Tristan1900 commented Nov 15, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Nov 15, 2024

tiprow bot commented Nov 15, 2024

codecov bot commented Nov 25, 2024 • edited Loading

Codecov Report

tiprow bot commented Feb 6, 2025

Tristan1900 commented Feb 6, 2025

tiprow bot commented Feb 6, 2025

Leavrth Feb 7, 2025

Choose a reason for hiding this comment

Leavrth Feb 7, 2025

Choose a reason for hiding this comment

Tristan1900 commented Feb 12, 2025

ti-chi-bot bot commented Feb 12, 2025

ti-chi-bot bot commented Feb 12, 2025

[LGTM Timeline notifier]

Tristan1900 commented Feb 12, 2025

Tristan1900 commented Nov 15, 2024 •

edited

Loading

codecov bot commented Nov 25, 2024 •

edited

Loading