execd: background command SSE stream intermittently cut by gvisor netstack before chunked terminator (peer closed / incomplete chunked read)

[xuxiong]

Component

components/execd (background command SSE stream) + sdks/sandbox/python (SSE client) + gvisor RuntimeClass

Environment

• OpenSandbox MCP server (Python SDK) sharing one httpx connection pool across concurrent sandbox SSE streams
• Sandboxes on gvisor RuntimeClass (user-space netstack) — confirmed via opensandbox-server log: Using Kubernetes RuntimeClass 'gvisor' for sandbox <id>
• execd version: v1.0.6 (confirmed via opensandbox-server log: execd_image=.../execd:v1.0.6). Note: v1.0.6 predates the signal-handling fix in 297f2ddb (shipped in v1.0.20), but the residual errors below are peer closed only, not EBADF/accept failures — see "Root cause certainty" below.
• 60 concurrent background: true commands against a single sandbox, 5 stress rounds

Summary

Background commands intermittently fail with RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read). The error is sporadic (0.3–0.8% per round, 1–2 commands per 60) and only affects background commands — foreground commands on the same sandbox never hit it. After enlarging the SDK keepalive pool (separate change) the rate dropped from ~3.7% to 0.3–0.8% but never reaches zero.

Reproduction

1. Start a sandbox on a gvisor-based RuntimeClass.
2. From the Python SDK, fire 60 concurrent sandbox.commands.run(..., background=True) calls against simple instant commands (date, expr, printf, sleep 1, ls -la, echo).
3. Repeat for 5 rounds.
4. Observe 1–2 RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read) per round, all on background commands, durations reported as 1–3.5s (background SSE streams should complete in <1s — the SDK is stuck waiting for the stream terminator).

Round	Errors	Rate	Notes
base	1/60	0.6%	all background
stress-1	2/60	0.3%	all background
stress-2	1/60	0.6%	all background
stress-3	2/60	0.3%	all background
stress-4	1/60	0.8%	all background

Foreground commands in the same rounds: 0 errors.

Symptom

incomplete chunked read means httpx/h11 was reading the chunked response body when the TCP connection was closed before the chunked terminator (0\r\n\r\n) arrived.

Background command SSE flow:

execd side (components/execd/pkg/runtime/command.go:286-390, runBackgroundCommand):
  1. OnExecuteInit  → emit init SSE event
  2. cmd.Start()    → start process (e.g. date)
  3. safego.Go(...) → async wait for process exit
  4. OnExecuteComplete → emit complete SSE event (synchronous, does not wait for process)
  5. return nil     → back to RunCommand handler
  6. waitForExecutionComplete → completeCh already closed, returns immediately
  7. time.Sleep(ApiGracefulShutdownTimeout)  // 1s, components/execd/pkg/web/controller/command.go:121
  8. handler returns → Go net/http sends chunked terminator 0\r\n\r\n

SDK side (sdks/sandbox/python/src/opensandbox/adapters/command_adapter.py:226-244):
  async with client.stream("POST", url) as response:
    async for event in aiter_sse_events(response):
      # reads init + complete events
    # continues iterating, waiting for stream end (chunked terminator)

The TCP connection is closed before step 8's terminator reaches the SDK.

Root cause (hypothesis — not fully verified)

Leading hypothesis: gvisor netstack probabilistically closes the TCP connection early during step 7 (time.Sleep(1s)) or while step 8 sends the terminator. Go net/http's terminator never reaches the SDK, and httpx/h11 raises incomplete chunked read.

Evidence supporting the hypothesis:

• gvisor RuntimeClass confirmed in opensandbox-server log (direct evidence of gvisor usage).
• execd v1.0.6 log shows no EBADF / accept failures in the residual-error rounds — only peer closed. This rules out execd-side active close via signal/fd corruption for these specific errors.
• SDK has no logic to close the connection mid-stream.
• The symptom pattern (short background SSE streams affected, long foreground streams unaffected) is consistent with user-space netstack connection reaping behavior.

What is NOT proven:

• No gvisor-side TCP close log was captured (gvisor runsc logs were not collected during the test).
• No A/B comparison against runc RuntimeClass was performed to confirm the error disappears without gvisor.
• The exact gvisor netstack mechanism (timer-based reap, fd table race in the sentry, etc.) is not identified.

Alternative explanations not ruled out:

• Go net/http on the execd side could be closing the connection due to some handler-level condition we did not identify.
• A Kubernetes/networking layer (CNI, kube-proxy) between the SDK and the Pod could be intervening.

Why only background commands

Dimension	background	foreground
SSE stream duration	<1s (init+complete emitted immediately)	equals command runtime (may be seconds)
Terminator window	sent after 1s sleep, narrow	sent after command ends, wide
Affected by early close	short connections more vulnerable	long-stable connections survive

Why retries: 0

The SSE client (_sse_client) uses unwrap_retry_transport to bypass RetryAsyncTransport (command_adapter.py:193) because SSE POST is non-idempotent. Even without bypass, RetryAsyncTransport only retries the transport layer (handle_async_request); body reading happens in AsyncClient.send after that, outside the retry scope.

Why early rounds fail more

base/stress-1 fail; stress-2..5 do not. Early rounds build new TCP connections (more races); later rounds reuse keepalive connections that are already stable. Enlarging the keepalive pool (separate change) cut the rate from ~3.7% to 0.3–0.8% by raising reuse, but reused connections can still be affected.

Impact

• Any high-concurrency client (MCP server, agent harness) issuing background commands against gvisor sandboxes sees sporadic 0.3–0.8% failures.
• Failures surface as RemoteProtocolError in the SDK; callers must treat them as transient and retry the whole command, but the SDK gives no signal that the command actually completed on the execd side (it did — complete was already emitted).

Suggested fixes

A. SDK: break on complete for background commands (smallest change, recommended)

In sdks/sandbox/python/src/opensandbox/adapters/command_adapter.py:235-244, break out of the SSE loop as soon as the complete event is received for background commands, instead of waiting for the chunked terminator. Semantically a background command is done once complete arrives; the terminator is redundant.

async for event in aiter_sse_events(response):
    event_node = _decode_sse_event_data(event.data)
    if event_node is None:
        continue
    await dispatcher.dispatch(event_node)
    if opts.background and event_node.get("type") == "complete":
        break  # do not wait for chunked terminator

~5 lines, SDK-only, no execd change. Note: async with client.stream() exit with an unread body marks the connection non-reusable (close, not keepalive) but does not raise.

B. execd: explicit flush before handler return

In components/execd/pkg/web/controller/command.go:121, after time.Sleep(ApiGracefulShutdownTimeout), explicitly flusher.Flush() and briefly wait (~100ms) so Go net/http writes the chunked terminator into the TCP buffer before the handler returns. Root-cause fix but requires an execd change and adds ~100ms latency.

C. MCP server: limited retry on empty SSE response

Retry RemoteProtocolError only when no SSE event was received (request did not take effect); do not retry if complete was already seen. Requires SDK to expose "did we see complete" state. Not recommended if A is adopted — A eliminates the error at the source.

D. gvisor → runc

Switching the sandbox RuntimeClass removes the user-space netstack race entirely but weakens isolation. Out of scope for this issue. Would also serve to verify the root cause hypothesis — if errors disappear under runc, the gvisor hypothesis is confirmed.

Recommendation

Adopt A (SDK break on complete) as the primary fix — minimal, correct, no execd change, and it sidesteps the root cause regardless of whether it is gvisor or something else. B is the execd-side root-cause fix if execd changes are acceptable. The keepalive-pool enlargement (already landed separately) stays as a general performance improvement but does not fully eliminate this race.

To fully confirm the root cause, an A/B test against runc RuntimeClass (option D, as a diagnostic) is recommended.

Other related information

• Related but distinct: egress: LLM SSE chunks >1MB truncated when TLS HTTP/1.1 upstream closes connection right after body #1461 (egress mitmproxy TLS truncation — same h11 incomplete chunked read symptom, different layer/root cause).
• execd version in test: v1.0.6. Current main branch is v1.0.22; the signal-handling fix (297f2ddb) shipped in v1.0.20. Upgrading may reduce other failure modes but is not expected to eliminate this specific peer closed error, since the symptom (chunked terminator loss) differs from signal/fd corruption (EBADF/accept failure).
• Background run design: feat(execd): support background runs in isolated sessions #1456.
• SSE parser fix (Unicode separators, unrelated): fix(python-sdk): preserve Unicode separators in SSE data #1444.