Skip to content

workerd-linux-64: Cap'n Proto IPC stream desynchronization (traversalLimitInWords exceeded) following high-volume internal DO RPC calls under Miniflare #7401

Description

@natew

Environment

  • workerd binary: @cloudflare/workerd-linux-64@1.20260722.1 (workerd 2026-07-22)
  • wrangler version: 4.114.0
  • miniflare version: 4.20260722.0
  • Host OS: Linux x86_64 (Ubuntu 26.04 LTS, kernel 6.8.0-40-generic)
  • Comparison Platform: macOS arm64 (workerd-darwin-arm64 under identical workload: 0 failures in 20 runs)
  • Production Infrastructure: Not reproduced on deployed Cloudflare edge infrastructure; tested and observed exclusively in local wrangler dev / Miniflare environments on Linux.

Summary

Under local wrangler dev (Miniflare) on Linux x86_64, executing a high volume of sequential internal Cap'n Proto RPC method calls on a Durable Object stub (~600 calls within a single incoming HTTP request) causes workerd-linux-64 to crash with a Cap'n Proto traversalLimitInWords violation when a subsequent HTTP request arrives immediately afterward.

The error occurs at an apparent framing desynchronization boundary on the Unix domain socket IPC connection between Miniflare and workerd, where the incoming message size is parsed as ~20.8 billion words (~167 GB), exceeding the 64 MB limit (options.traversalLimitInWords = 8388608).

Under an interleaved test on Linux (20 runs per arm):

  • Baseline Arm (subsequent request issued immediately at 0ms delay to a valid DO RPC endpoint): 4 crashes out of 20 runs (20% failure rate).
  • Delay Control Arm (identical workload with a 1000ms delay before the subsequent request): 0 crashes out of 20 runs (0% failure rate).
  • Comparison Platform (identical workload on macOS arm64): 0 crashes out of 20 runs (0% failure rate).

Crash Output & Logs

stderr

NOSENTRY RPC connection broken for non-DISCONNECTED reason.; exception = capnp/serialize-async.c++:789: failed: expected expectedSizeInWords <= options.traversalLimitInWords [20896653011 <= 8388608]; incoming RPC message exceeds size limit
stack: node_modules/@cloudflare/workerd-linux-64/bin/workerd@356db05 node_modules/@cloudflare/workerd-linux-64/bin/workerd@356cc32 node_modules/@cloudflare/workerd-linux-64/bin/workerd@356cb86 node_modules/@cloudflare/workerd-linux-64/bin/workerd@35704bb node_modules/@cloudflare/workerd-linux-64/bin/workerd@357007e node_modules/@cloudflare/workerd-linux-64/bin/workerd@3571d79 node_modules/@cloudflare/workerd-linux-64/bin/workerd@35728a0 node_modules/@cloudflare/workerd-linux-64/bin/workerd@356f9ee node_modules/@cloudflare/workerd-linux-64/bin/workerd@3564024 node_modules/@cloudflare/workerd-linux-64/bin/workerd@3564cb8 node_modules/@cloudflare/workerd-linux-64/bin/workerd@3564998 node_modules/@cloudflare/workerd-linux-64/bin/workerd@356b461 node_modules/@cloudflare/workerd-linux-64/bin/workerd@3568853 node_modules/@cloudflare/workerd-linux-64/bin/workerd@3568936 node_modules/@cloudflare/workerd-linux-64/bin/workerd@3568a52 node_modules/@cloudflare/workerd-linux-64/bin/workerd@3568b20
kj/async-io-unix.c++:186: disconnected: ::write(fd, buffer.begin(), buffer.size()): Broken pipe

Miniflare client error

500: Error: Network connection lost

Stack Frames

The failure occurs in Cap'n Proto stream deserialization:

  • File: capnp/serialize-async.c++:789
  • Condition: expectedSizeInWords <= options.traversalLimitInWords (20896653011 <= 8388608)
  • Stack path: BufferedMessageStream::tryReadMessageImpl reading expectedSizeInWordsFromPrefix. The computed size (20,896,653,011 words = ~167 GB) indicates stream alignment corruption where arbitrary payload bytes are read as the Cap'n Proto segment length prefix.

Decisive Evidence: IPC Layer vs Durable Object Handler

To isolate whether the crash was caused by unawaited async work, memory pressure, or SQLite storage state inside the Durable Object actor, we tested a separate isolating arm:

  1. Request 1 executes the ~600 internal RPC batch calls on the DO stub as usual.
  2. Request 2 is issued immediately (0ms delay), but targets an invalid endpoint that the Worker entrypoint rejects with HTTP 400 Bad Request prior to calling env.DO.get() or dispatching any calls to the Durable Object.

Result in this separate 400-rejection arm: 5 crashes out of 20 runs (25% failure rate) with the identical Cap'n Proto exception and broken pipe.

Because Request 2 is rejected at the worker entrypoint and never touches the Durable Object, the crash cannot be attributed to the Durable Object application code or SQLite storage state. It points directly to the underlying IPC / Unix domain socket stream framing layer between the Miniflare process and workerd.

Workarounds Observed

  1. Introducing a 1000ms pause between Request 1 and Request 2 completely eliminated the crash (20/20 PASS).
  2. Issuing an intermediate HTTP probe to the DO (e.g. await stub.fetch(...)) before making direct RPC method calls also eliminated the crash (20/20 PASS), suggesting that cycling the actor event loop turn allows pending socket buffers to drain.

Reproduction Status

We have not yet reduced this to a minimal 10-line standalone script outside our test harness. The boundary conditions required to reproduce:

  1. Environment: Linux x86_64, workerd-linux-64@1.20260722.1, wrangler dev (Miniflare 4.20260722.0).
  2. Workload: An initial HTTP request that triggers approximately 600 sequential Cap'n Proto RPC method calls (stub.someMethod(...)) on a Durable Object stub.
  3. Timing: Issuing any subsequent HTTP request to the worker immediately (0ms delay) upon completion of the first request.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions