Repository navigation
HBASE-30348 Throttles should work for system tables (excluding quotas, meta, namespace) - #8774
Draft
rmdmattingly wants to merge 1 commit into
Draft
rmdmattingly wants to merge 1 commit into
rmdmattingly wants to merge 1 commit into
Conversation
…, meta, namespace) System tables outside the hbase namespace, such as the backup tables, can generate a lot of IO but could never be throttled. Only hbase:meta, hbase:quota and hbase:namespace remain exempt, because they are read while the master is still initializing. Throttling is now best effort: no request waits for the quota cache to load. Until a load fully succeeds, requests get default quota state, which does not throttle, and failed loads are retried. Waiting on the load inline is what deadlocked master startup on branch-2 in the first attempt at this change, since the load needs hbase:quota and the master. Each refresh also reloads once more when it finds tables with new quotas. Cluster scope table quotas are converted to machine scope using factors computed before the fetch, so a newly limited table previously got the full cluster limit on every RegionServer until the next refresh. Per-user table quotas now get factors as well; before, only tables with a table quota did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HBASE-30348
Draft to get CI on the direction discussed in #8617: throttling is best effort, so no request ever waits for the quota cache, and a request that arrives before quota data is available is simply not throttled.
Changes from #8617
The initial quota cache load never blocks a request. #8617 bounded the wait with a timeout, but every request still serialized on the initializer lock and waited out that timeout while the master was initializing. Now the first lookup starts the load in the background and returns default quota state immediately. That removes the deadlock regardless of which tables are read during master initialization, so the timeout config is gone.
The cache is only ready once a load fully succeeds. Failed loads are retried every second. Otherwise a load that ran before
hbase:quotaexisted, which is the branch-2 startup case, would mark the cache ready with nothing in it and leave quotas unenforced until the next periodic refresh, 12 hours by default.Quota factors for newly limited tables. Making the load asynchronous exposed an existing bug that
TestClusterScopeQuotaThrottlehad only been passing because of the extra synchronous load. Cluster scope table quotas are converted to machine scope using factors computed from the quotas cached before the fetch, so a table that just got a quota was given factor 1, the full cluster limit on every RegionServer, until the next refresh. A refresh that finds tables with new quotas now recomputes factors and fetches again. Per-user table quotas now get factors too; previously only tables with a table quota did.The
hbase:meta,hbase:quotaandhbase:namespaceexemptions are unchanged here. With the load no longer blocking, they are no longer what prevents the startup deadlock, so whether to keep them is a separate question.Testing
TestQuotaCache2adds two tests: a lookup returns within a bounded time while thehbase:quotaread hangs, and a failed initial load is retried until it succeeds.TestQuotaCachenow waits for the initial load before asserting.Locally on master with JDK 17,
org.apache.hadoop.hbase.quotas: 303 tests, 0 failures, 0 errors, 18 skipped (TestQuotaThrottleis disabled by HBASE-27243).🤖 Generated with Claude Code