[ISSUE #479]fix: duplicate pulls due to concurrency issues in rebalance. #480
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fix #479
根因分析
根因:AsyncPullCallback 构建存在并发问题,导致多个 pull 线程消费同一个队列。
通过以上 5 个条件:如果两次 Rebalance 相隔时间很短,mq 对应的之前的 PullRequest 还来不及因设置为 drop 而结束当前的拉取。第二次 rebalance 就替换掉了 AsyncPullCallback 中的局部变量 PullRequest,导致上一次的拉取请求又拿到了非 drop 的 pullRequest。此时就会出现 2 个 pullRequest 同时拉取同一个队列的情况。依次类推,如果重复多次短时间 rebalance,单个分区可能存在多个 pullRequest 同时拉取。
新的 pullRequest 替换掉旧的 pullRequest, drop 状态被配置成了 false
![image](https://private-user-images.githubusercontent.com/50660789/392204578-632334cc-c3b8-4ab1-afe0-69dbea70c510.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkzODI4NjgsIm5iZiI6MTczOTM4MjU2OCwicGF0aCI6Ii81MDY2MDc4OS8zOTIyMDQ1NzgtNjMyMzM0Y2MtYzNiOC00YWIxLWFmZTAtNjlkYmVhNzBjNTEwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjEyVDE3NDkyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTUzNzMxMmRjMzJlZGQwZjM5ZTc0NmU3ZTA3MzgyMzc1OGEyMWQ0ZGFiMDllMzY1NjUxNDJkMWMzMjRjNzA4YzUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.hzRlTSrJP1oU9SHSZGsbebDj2a9hyytaQQTnk4BqqtE)
![image](https://private-user-images.githubusercontent.com/50660789/392204819-6f13b9ce-402d-4db9-b059-b5d6b7998650.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkzODI4NjgsIm5iZiI6MTczOTM4MjU2OCwicGF0aCI6Ii81MDY2MDc4OS8zOTIyMDQ4MTktNmYxM2I5Y2UtNDAyZC00ZGI5LWIwNTktYjVkNmI3OTk4NjUwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjEyVDE3NDkyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY4NzQ0MDcyNzZmMDExNjU2ZGRlYzUzNGJlMWYxNzk4Mzk1MGE1MGVkNjcwZWZkMmE4ZmU1ZjIwOGJiNTc2MjgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.VVZmMz81ayhJPsy_5KikZaujUYTcwd3xF1wi3m_NtVE)
上次还没处理结束的拉取线程因为状态变化不能正常退出