transport: drain the control buffer in batches to reduce lock contention

[arjan-bal]

Today, loopy acquires controlBuffer.mu once for every control frame it reads. Stream goroutines acquire the same mutex in put() for every frame they queue, so the lock is constantly handed back and forth between loopy and the producers, and both sides frequently find it held.

This PR makes loopy take all the queued frames in a single critical section. When its local batch is empty, get() swaps controlBuffer.list with an empty, loopy-owned consumerList (no allocation, since the two ring buffers just trade places) and then returns frames from consumerList one at a time without holding the lock. Loopy's lock acquisitions drop from one per frame to one per batch.

Loopy still calls get() for each frame and calls processData() after handling it, so writes remain round-robin across streams.

Throttling

The number of queued transport response frames is used to throttle the reader (throttle()). With batching, frames leave the control buffer's list in bulk, but should only count as consumed once loopy dequeues them from its batch.

• transportResponseFrames is now an atomic.Int64 counting the frames in both list and consumerList. Producers increment it while holding mu; loopy decrements it without holding mu.
• The throttling channel is created or closed by updateThrottleLocked(), which is called while holding mu whenever the count crosses maxQueuedTransportResponseFrames. Loopy only acquires the lock for a downward crossing, which is rare.
• updateThrottleLocked() acts on the current count rather than on the direction of the crossing that triggered it. Because the consumer decrements without holding the lock, the count may have crossed the threshold again by the time the update runs. A naive edge-triggered version (create on the upward crossing, close on the downward crossing) can replace a channel that a reader is blocked on without closing it, leaving the reader stuck in throttle(). Being idempotent and serialized by mu, the last update always runs after the last crossing and leaves the channel in the correct state.

Shutdown

On the server, finish() can be called from Close() on a different goroutine than loopy, so it cannot touch loopy's consumerList. Loopy cleans up any unhandled frames in its batch via discardConsumerList() after calling finish(), using the same cleanup logic (orphaning clientHeaders, freeing data of unprocessed dataFrames). closed is now an atomic.Bool (still only written under mu) so that get() can stop handing out frames from the batch once the control buffer is finished, matching the existing behavior.

Benchmarks

benchmain with -bufconn -networkMode=Local -compression=off -benchtime=60s, GOMAXPROCS=4. Each round runs master and this PR at the same time, pinned to separate sets of 4 physical cores (using taskset), alternating core sets between rounds.

Unary, 100 concurrent calls, 0-byte payloads

Round	QPS	p50 latency	Avg latency	p99 latency
1	+7.10%	-8.47%	-6.62%	-2.02%
2	+7.28%	-8.60%	-6.76%	-2.79%
3	+6.42%	-7.44%	-6.00%	-3.12%
Mean	+6.93%	-8.17%	-6.46%	-2.64%

Streaming, 200 concurrent calls, 100-byte payloads

Round	QPS	p50 latency	Avg latency	p99 latency
1	+4.21%	-4.70%	-4.15%	-1.54%
2	+4.33%	-4.86%	-4.16%	-0.94%
3	+1.04%	-1.70%	-0.93%	+1.23%
Mean	+3.19%	-3.75%	-3.08%	-0.42%

Bytes/op and allocs/op are unchanged (within ±0.05%) in all rounds.

RELEASE NOTES:

• transport: reduce lock contention between stream goroutines and the transport writer by draining the control buffer in batches, improving throughput.

[codecov]

Codecov Report

❌ Patch coverage is 97.61905% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.72%. Comparing base (39ebc82) to head (646ce45).
⚠️ Report is 16 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/transport/controlbuf.go	97.61%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #9499   +/-   ##
=======================================
  Coverage   87.71%   87.72%           
=======================================
  Files         436      436           
  Lines       30822    30837   +15     
=======================================
+ Hits        27036    27052   +16     
+ Misses       3785     3784    -1     
  Partials        1        1           

Files with missing lines	Coverage Δ
internal/transport/controlbuf.go	94.90% <97.61%> (-0.57%)	⬇️

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[easwars]

@arjan-bal

How did you run into this?

Is any user/customer seeing this being a problem in practice?

[arjan-bal]

Hi Easwar, I was testing another controlbuf improvement (#9444, now merged), which I expected to improve QPS by only a couple of percent, since controlbuf wasn't prominent in the flame graphs. It turned out to have a much larger impact: roughly 10–15% higher QPS in local benchmarks and about 5% in the k8s benchmarks.

That suggested controlbuf is a bigger bottleneck than the profiles show, and that optimizing it pays off more than expected. So I used Claude Opus to analyze the profiles from a local benchmark run alongside the transport code, looking for other areas to improve. One of its suggestions was to batch controlbuf drain operations, so that their cost is amortized instead of scaling with the number of streams. Based on the profiles, it estimated a 1–2% QPS gain. Once again, the measured improvement was much larger: around 8% on top of #9444.

Is any user/customer seeing this being a problem in practice?

I'm not aware of a specific user report. Since this overhead doesn't stand out in profiles, it's unlikely that a user could trace a performance problem back to this code path. Still, users regularly benchmark gRPC Go against other RPC frameworks and against REST clients when picking a library for their services. Combined with the already-merged #9444, this change brings a roughly 10–20% improvement in gRPC Go unary performance in a single release, which directly helps how gRPC Go compares in such evaluations.