Wrap fatal TX errors in a new vterrors code

[frouioui]

Description

This Pull Request adds a new vterrors code (VT15001) that wraps common errors appearing when a cluster event (rollout, PRS, primary shutdown, ...) happens on a shard with an active transaction.

The errors listed in #17668 now look like this:

target: ks.0.primary: vttablet: rpc error: code = FailedPrecondition desc = VT15001: transient error, please retry the transaction: wrong tablet type: PRIMARY, want: REPLICA or [] (errno 1105) (sqlstate HY000) during query: insert into vt_insert_test(id, msg) values (162, 'test 162')

target: ks.0.primary: VT15001: transient error, please retry the transaction: vttablet: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:7108: connect: connection refused" (errno 1105) (sqlstate HY000) during query: insert into vt_insert_test(id, msg) values (650, 'test 650')

target: ks.0.primary: VT15001: transient error, please retry the transaction: tablet: cell:"zone1" uid:101 is either down or nonexistent (errno 1105) (sqlstate HY000) during query: insert into vt_insert_test(id, msg) values (1029, 'test 1029')

In addition to wrap these errors, we also make sure to rollback and clear the transaction (if applicable) when such error happens, leaving the client responsible for catching the error and retrying the transaction.

Documentation

• Update error code documentation (#17669) website#1929

Related Issue(s)

• Closes Enable applications to detect stalled connections during PRS #17668

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[codecov]

Codecov Report

Attention: Patch coverage is 71.71717% with 28 lines in your changes missing coverage. Please review.

Project coverage is 67.45%. Comparing base (70114ad) to head (5baee4d).
Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
go/vt/vterrors/vterrorsgen/main.go	0.00%	10 Missing ⚠️
go/vt/vtgate/plan_execute.go	61.11%	7 Missing ⚠️
go/vt/vttablet/queryservice/wrapped.go	90.62%	3 Missing ⚠️
go/vt/vterrors/code.go	87.50%	2 Missing ⚠️
go/vt/vtgate/executor.go	50.00%	2 Missing ⚠️
go/vt/vtgate/scatter_conn.go	33.33%	2 Missing ⚠️
go/vt/vtgate/tx_conn.go	60.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17669      +/-   ##
==========================================
- Coverage   67.94%   67.45%   -0.50%     
==========================================
  Files        1586     1592       +6     
  Lines      255224   258250    +3026     
==========================================
+ Hits       173420   174198     +778     
- Misses      81804    84052    +2248     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[deepthi]

"session invalidated" may be misleading

• if you are in a transaction, that needs to be closed, and a new transaction opened.
• if you are not in a transaction, a retry is probably what you need to do.
  @harshit-gangal wdyt?

[harshit-gangal]

It is the transaction which no longer exists. They can continue to use the same connection.
"rollback", followed by a retry is fine.
outside of transaction, retry is enough as Deepthi pointed.
All 3 error message should have different error code as they happen at different point in time of PRS/ERS

[deepthi]

Notes from further offline discussion:

• In order to close the transaction, you have to issue a COMMIT or a ROLLBACK from the client. Because the vtgate-level transaction is effectively pinned to the primary tablets of participating shards, these will also produce errors
• An idea to mitigate these is for us to close the transaction at the vtgate level so that the ROLLBACK becomes a no-op instead of resulting in an error.

We need to spend some more time exploring these options.

[frouioui]

If there is a cluster event error, we will invalidate the session, the client is supposed to catch the error and issue a rollback, until then commit and other queries will fail until a rollback is received.

[harshit-gangal]

nit: let's make it clear, options:

1. freeze_until_rollback
2. fail_until_rollback
3. block_until_rollback
4. rollback_expected
5. lock_until_rollback
6. rollback_required
7. reject_until_rollback
8. error_until_rollback

[frouioui]

I changed it via 812df8f, I picked error_until_rollback.

[harshit-gangal]

instead of changing in this file, we can do it in go/vt/vttablet/queryservice/wrapped.go
This is what scatter conn uses.

[harshit-gangal]

one step further, we can check the error in scatter_conn itself.

[frouioui]

I moved the wrapping to the wrapper via f5ee299

[harshit-gangal]

I think we should validate by selecting from the table that no data exists.

[frouioui]

Issue with selecting is that the commit can be a partial commit, leading to rows on the healthy shards.

[harshit-gangal]

StreamExecute calls this only when it is called for DML query. This is valid for Select query as well inside a transaction

		// Check if there was partial DML execution. If so, rollback the effect of the partially executed query.
		if err != nil {
			if !canReturnRows(plan.Type) {
				return e.rollbackExecIfNeeded(ctx, safeSession, bindVars, logStats, err)
			}
			return err
		}

[harshit-gangal]

Can we handle this at the scatter_conn level by deciding to rollback the shard session on a VT15001 error?
Otherwise, the responsibility falls on the callers of scatter_conn methods (Execute, ExecuteMultiShard, and StreamExecuteMulti).

This behavior should be independent of the executed query—it’s purely about whether the transaction is open or not.