fix(sqllab): roll back session before retrying get_query after a broken transaction - #42675
Conversation
…en transaction `get_query` catches any exception from the ORM lookup and relies on the `backoff` decorator to retry up to 5 times. When the underlying failure is (or causes) a SQLAlchemy `PendingRollbackError` - e.g. a `PendingRollbackError` chained under an `OperationalError`/`QueryCanceled` from a dropped connection or statement timeout - the session is left in a broken state that SQLAlchemy refuses to use again until `.rollback()` is called explicitly. Since the session was never rolled back, every one of the 5 retries reused the same poisoned session and failed identically, so the retry loop never had a chance to recover from what may be a transient connection blip. Roll back the session in the except block before raising `SqlLabException` so each `backoff` retry starts from a clean session. The exception raised and logged is unchanged. Fixes SUPERSET-PYTHON-WDZ Co-Authored-By: Claude <noreply@anthropic.com>
Code Review Agent Run #272215Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #42675 +/- ##
==========================================
- Coverage 65.43% 65.43% -0.01%
==========================================
Files 2810 2810
Lines 159460 159475 +15
Branches 36396 36398 +2
==========================================
+ Hits 104350 104354 +4
- Misses 53068 53076 +8
- Partials 2042 2045 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![codeant-ai-for-open-source[bot]](https://avatars.githubusercontent.com/u/151810221?s=60&v=4){kind=small}
| except Exception as ex: | ||
| # roll back so a poisoned session (e.g. PendingRollbackError after a | ||
| # failed flush) doesn't fail every subsequent backoff retry identically | ||
| db.session.rollback() |
There was a problem hiding this comment.
Suggestion: A failed database rollback can itself raise an exception, which would replace the original error and escape as a non-SqlLabException. Because the decorator retries only SqlLabException, this prevents retries precisely when the session or connection is too broken to roll back normally. Preserve the original wrapped exception and ensure rollback failures do not bypass the retry contract. [possible bug]
Severity Level: Major ⚠️
- ❌ Broken database connections can bypass `get_query()` retries.
- ⚠️ SQL Lab error handling receives an unexpected rollback exception.
- ⚠️ Transient query failures may fail instead of recovering.
(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** superset/sql_lab.py
**Line:** 166:166
**Comment:**
*Possible Bug: A failed database rollback can itself raise an exception, which would replace the original error and escape as a non-`SqlLabException`. Because the decorator retries only `SqlLabException`, this prevents retries precisely when the session or connection is too broken to roll back normally. Preserve the original wrapped exception and ensure rollback failures do not bypass the retry contract.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
|
The flagged issue is correct. If except Exception as ex:
try:
db.session.rollback()
except Exception:
# Log rollback failure if necessary
pass
raise SqlLabException("Failed at getting query") from exsuperset/sql_lab.py |
…ntract get_query's backoff decorator only retries on SqlLabException. If db.session.rollback() itself raises (e.g. the connection is fully dead), that new exception would replace the intended SqlLabException and bypass the retry contract. Swallow rollback failures so the original lookup error is always what gets raised. Co-Authored-By: Claude <noreply@anthropic.com>
Code Review Agent Run #974632Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
2 participants
Sentry issue
SUPERSET-PYTHON-WDZ —
SqlLabException: Failed at getting query— 30,500 occurrences, 24 users impacted, first seen 2025-04-05, still firing daily in production.Root cause
superset/sql_lab.py::get_queryis wrapped in abackoff.on_exception(SqlLabException, interval=1, max_tries=5)retry loop:The Sentry event's exception chain shows a SQLAlchemy
PendingRollbackError("This Session's transaction has been rolled back due to a previous exception during flush... first issue Session.rollback()"), itself chained under anOperationalError/QueryCanceledfrom a dropped connection or statement timeout. Once a session enters this state, SQLAlchemy refuses any further operation on it until.rollback()is called explicitly.Because
get_query'sexceptblock never rolls back, every one of the 5backoffretries reuses the same poisoned session and fails identically with the samePendingRollbackError— the retry loop is guaranteed to burn all 5 attempts and give up, with zero chance to recover from what may have been a transient connection blip.Verified with a standalone repro (pinned
sqlalchemy==1.4.54, in-memory sqlite): poisoning a session via a failed flush, then calling aget_query-shaped function under the samebackoff.on_exceptiondecorator — without a rollback, all 5 retries fail identically; addingsession.rollback()in the except block lets the very next retry succeed.Fix
Add
db.session.rollback()inget_query'sexceptblock before re-raisingSqlLabException, so eachbackoffretry starts from a clean session and has a real chance of succeeding instead of a guaranteed-failure loop.Tradeoffs
None — this is additive-only. The exception raised (
SqlLabException) and logged (viaget_query_backoff_handler/get_query_giveup_handler) is unchanged; only the session state between retries changes, giving the existing retry mechanism a real chance to work as originally designed. No change toexecute_query/execute_sql_statements/handle_query_error, and no change to thebackoffdecorator's parameters.Testing
test_get_query_rolls_back_session_before_retryingtotests/unit_tests/sql_lab_test.py, assertingget_queryrecovers via retry after a first failure, and thatdb.session.rollback()is called before the successful retry.tests/unit_tests/sql_lab_test.py— 12 passed.ruff check/ruff format --check(pinned0.9.7) — clean on both changed files.Shortcut
sc-115684
Fixes SUPERSET-PYTHON-WDZ
🤖 Generated with Claude Code