Skip to content

fix(http2): Http2Client evicts dead connections from its pool and keeps healthy ones on a stream reset - #2010

Draft
mosuem wants to merge 1 commit into
mosum/http2-5b-buffered-sink-done-after-write-failurefrom
mosum/http2-6-http2-client-pool-and-flow-control
Draft

mosuem wants to merge 1 commit into
mosum/http2-5b-buffered-sink-done-after-write-failurefrom
mosum/http2-6-http2-client-pool-and-flow-control

Conversation

@mosuem

@mosuem mosuem commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Note

This PR was generated by an AI coding agent (Jetski) on behalf of @mosuem.

Summary

Two pool-health bugs in Http2Client, one hiding the other:

  1. A dead connection stayed in the pool. When a request failed because its connection died (socket closed, GOAWAY + close, protocol error), the stream's onError callback called lease.release() first, and only afterwards did send()'s attempt() catch block call lease.markFailed(). release() is where ClientPool._closeIfIdle runs, and at that moment createFailed was still false, so the connection was kept as the pool's idle connection. The later markFailed() only set a flag on an already-released slot, so nothing ever closed it: it stayed in _connections (counted by connectionCount, skipped by _select) until close(). The same happened when a connection died under a response body.
  2. A healthy connection was marked failed on any stream error. attempt() marked the lease failed on every error, including a server resetting just that one stream (RST_STREAM, RFC 9113 Section 5.4.2 — a stream error "does not affect the other streams on the connection"). Together with (1) that meant every reset stream left a zombie connection behind and the next request dialed a new one.

Changes

  • lib/src/client_pool.dart: PoolLease.markFailed() now works regardless of ordering: if the slot was already released, it evicts the (idle) connection right away via _closeIfIdle, which is made idempotent (_connections.remove is checked) so a lease that was released and later marked failed cannot close a connection twice. Fixing this in the pool rather than at each call site means no caller has to get the markFailed()/release() order right.
  • lib/src/http2_client.dart: a single _isConnectionFailure(connection, error) predicate — TransportConnectionException, _ConnectionClosedByPeer, or !connection.isOpen — decides whether a failure condemns the connection. It is used in the stream's onError (so a connection that dies under a body is evicted at once, not on the next request) and in attempt()'s catch block (so a reset of one stream no longer evicts the connection).

Not changed, as a candidate follow-up: ClientConnection.isOpen is also false while the connection is merely at the peer's SETTINGS_MAX_CONCURRENT_STREAMS limit, so in that narrow state a healthy connection could still be evicted (gracefully: it is closed once idle and a new one is dialed). A dedicated "finishing or terminated" getter would make the predicate exact; _sendOverHttp2's existing !transport.isOpen check has the same property.

Test Verification (Fails Before $\rightarrow$ Passes After)

  • a-lease-failed-after-its-release-evicts-the-connection in test/client_pool_test.dart (pool-level, with the mock connections).
  • evicts-a-dead-connection-but-keeps-one-whose-stream-was-reset in test/http2_client_test.dart: request 1 has its connection terminated before any response (pool must be empty afterwards), request 2 gets RST_STREAM on a fresh connection (pool must still hold it), request 3 must be served on that same connection. No sleeps; every assertion follows the request's own completion. Also checked that reverting only the attempt() guard fails this test at the "connection kept" assertion (Expected: <1> Actual: <0>).
  • releases-the-slot-when-the-response-body-errors (existing) now also asserts connectionCount == 0 after a connection dies under a body.

Before fix (tests on the previous library code):

00:00 +0 -1: test/client_pool_test.dart: client-pool-test a-lease-failed-after-its-release-evicts-the-connection [E]
  Expected: <0>
    Actual: <1>
00:00 +0 -2: test/http2_client_test.dart: http2-client-test releases-the-slot-when-the-response-body-errors [E]
  Expected: <0>
    Actual: <1>
00:00 +0 -3: test/http2_client_test.dart: http2-client-test evicts-a-dead-connection-but-keeps-one-whose-stream-was-reset [E]
  Expected: <0>
    Actual: <1>

After fix:

00:00 +3: All tests passed!

Full package:http2 suite at this point of the stack (on master): +267 ~7: All tests passed!

@mosuem
mosuem added this pull request to stack #2011 October 8, 2026 08:32
@mosuem
mosuem force-pushed the mosum/http2-6-http2-client-pool-and-flow-control branch from f8bbcfd to 3b80204 Compare October 8, 2026 10:36
@mosuem mosuem changed the title fix(http2): evict dead connections on pre-header failure, keep healthy connections on stream reset, and raise Http2Client receive windows fix(http2): Http2Client evicts dead connections from its pool and keeps healthy ones on a stream reset Oct 8, 2026
@mosuem
mosuem removed this pull request from stack #2011 October 8, 2026 10:38
@mosuem
mosuem changed the base branch from mosum/http2-5-frame-reader-cancel-and-goaway-exception to mosum/http2-5b-buffered-sink-done-after-write-failure October 8, 2026 10:38
@mosuem
mosuem added this pull request to stack #2016 October 8, 2026 10:42
@mosuem
mosuem force-pushed the mosum/http2-6-http2-client-pool-and-flow-control branch from 3b80204 to faab4e8 Compare October 8, 2026 12:04
@mosuem
mosuem force-pushed the mosum/http2-6-http2-client-pool-and-flow-control branch from faab4e8 to b4bd763 Compare October 9, 2026 07:54
@mosuem
mosuem removed this pull request from stack #2016 October 9, 2026 07:55
@mosuem
mosuem added this pull request to stack #2020 October 9, 2026 07:55
@mosuem
mosuem force-pushed the mosum/http2-6-http2-client-pool-and-flow-control branch from b4bd763 to c157584 Compare October 9, 2026 08:18
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

PR Health

Coverage ✔️
File Coverage
pkgs/http2/lib/src/client_pool.dart 💚 100 %
pkgs/http2/lib/src/http2_client.dart 💚 89 % ⬆️ 0 %

This check for test coverage is informational (issues shown here will not fail the PR).

This check can be disabled by tagging the PR with skip-coverage-check.

License Headers ✔️
// Copyright (c) 2026, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.
Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers
Files
pkgs/http_multi_server/test/cert.dart

This check can be disabled by tagging the PR with skip-license-check.

Breaking changes ✔️
Package Change Current Version New Version Needed Version Looking good?
http2 Non-Breaking 3.1.0 3.2.0-wip 3.2.0-wip ✔️

This check can be disabled by tagging the PR with skip-breaking-check.

Unused Dependencies ✔️
Package Status
http2 ✔️ All dependencies utilized correctly.

For details on how to fix these, see dependency_validator.

This check can be disabled by tagging the PR with skip-unused-dependencies-check.

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package Leaked API symbol Leaking sources

This check can be disabled by tagging the PR with skip-leaking-check.

Changelog Entry ✔️
Package Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

…ps healthy ones on a stream reset

> [!NOTE]
> This PR was generated by an AI coding agent (Jetski) on behalf of @mosuem.

### Summary

Two pool-health bugs in `Http2Client`, one hiding the other:

1. **A dead connection stayed in the pool.** When a request failed because its connection died (socket closed, `GOAWAY` + close, protocol error), the stream's `onError` callback called `lease.release()` first, and only afterwards did `send()`'s `attempt()` catch block call `lease.markFailed()`. `release()` is where `ClientPool._closeIfIdle` runs, and at that moment `createFailed` was still `false`, so the connection was kept as the pool's idle connection. The later `markFailed()` only set a flag on an already-released slot, so nothing ever closed it: it stayed in `_connections` (counted by `connectionCount`, skipped by `_select`) until `close()`. The same happened when a connection died under a response body.
2. **A healthy connection was marked failed on any stream error.** `attempt()` marked the lease failed on *every* error, including a server resetting just that one stream (`RST_STREAM`, RFC 9113 Section 5.4.2 — a stream error "does not affect the other streams on the connection"). Together with (1) that meant every reset stream left a zombie connection behind and the next request dialed a new one.

### Changes

- **`lib/src/client_pool.dart`**: `PoolLease.markFailed()` now works regardless of ordering: if the slot was already released, it evicts the (idle) connection right away via `_closeIfIdle`, which is made idempotent (`_connections.remove` is checked) so a lease that was released and later marked failed cannot close a connection twice. Fixing this in the pool rather than at each call site means no caller has to get the `markFailed()`/`release()` order right.
- **`lib/src/http2_client.dart`**: a single `_isConnectionFailure(connection, error)` predicate — `TransportConnectionException`, `_ConnectionClosedByPeer`, or `!connection.isOpen` — decides whether a failure condemns the connection. It is used in the stream's `onError` (so a connection that dies under a body is evicted at once, not on the next request) and in `attempt()`'s catch block (so a reset of one stream no longer evicts the connection).

Not changed, as a candidate follow-up: `ClientConnection.isOpen` is also `false` while the connection is merely at the peer's `SETTINGS_MAX_CONCURRENT_STREAMS` limit, so in that narrow state a healthy connection could still be evicted (gracefully: it is closed once idle and a new one is dialed). A dedicated "finishing or terminated" getter would make the predicate exact; `_sendOverHttp2`'s existing `!transport.isOpen` check has the same property.

### Test Verification (Fails Before $\rightarrow$ Passes After)

- `a-lease-failed-after-its-release-evicts-the-connection` in `test/client_pool_test.dart` (pool-level, with the mock connections).
- `evicts-a-dead-connection-but-keeps-one-whose-stream-was-reset` in `test/http2_client_test.dart`: request 1 has its connection terminated before any response (pool must be empty afterwards), request 2 gets `RST_STREAM` on a fresh connection (pool must still hold it), request 3 must be served on that same connection. No sleeps; every assertion follows the request's own completion. Also checked that reverting only the `attempt()` guard fails this test at the "connection kept" assertion (`Expected: <1> Actual: <0>`).
- `releases-the-slot-when-the-response-body-errors` (existing) now also asserts `connectionCount == 0` after a connection dies under a body.

**Before fix** (tests on the previous library code):
```text
00:00 +0 -1: test/client_pool_test.dart: client-pool-test a-lease-failed-after-its-release-evicts-the-connection [E]
  Expected: <0>
    Actual: <1>
00:00 +0 -2: test/http2_client_test.dart: http2-client-test releases-the-slot-when-the-response-body-errors [E]
  Expected: <0>
    Actual: <1>
00:00 +0 -3: test/http2_client_test.dart: http2-client-test evicts-a-dead-connection-but-keeps-one-whose-stream-was-reset [E]
  Expected: <0>
    Actual: <1>
```

**After fix:**
```text
00:00 +3: All tests passed!
```
Full `package:http2` suite at this point of the stack (on `master`): `+267 ~7: All tests passed!`
@mosuem
mosuem force-pushed the mosum/http2-6-http2-client-pool-and-flow-control branch from c157584 to 9174e50 Compare October 9, 2026 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant