Skip to content

spawn: send a sliced Bun.file() at stdin in chunks of its window - #44301

Draft
robobun wants to merge 2 commits into
robobun/b87007c8/blob-slice-fd-consumersfrom
robobun/b87007c8/stdin-file-window
Draft

robobun wants to merge 2 commits into
robobun/b87007c8/blob-slice-fd-consumersfrom
robobun/b87007c8/stdin-file-window

Conversation

@robobun

@robobun robobun commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • Bun.spawn({ stdin: Bun.file(p).slice(10, 20) }) sends the child the whole file. So do spawnSync, new Response(slice), slice.stream() and the shell.
  • Stdio::extract_blob (src/runtime/api/bun/spawn/stdio.rs:565) turns a file Blob into Stdio::Path or Stdio::Fd. Neither holds the window.

Fix

  • extract_blob opens the window in the parent before a child exists, and reads the first chunk.
  • Stdio::FileWindow carries it to the StaticPipeWriter. After each chunk the writer reads the next (at most 64 KiB) and waits for a writable event.
  • Verified: test/js/bun/spawn/spawn.test.ts (43 cases, 32 fail on base). Self-reviewed: 9 concerns raised, 4 addressed (Notes).

Background

Downsides

  • New errors: a window of a missing file throws ENOENT (main creates the file). A window at stdio[3] or above throws (main: the whole file).
  • The child's fd 0 is a socket, so /dev/stdin gives ENXIO. A parent that exits after unref() sends 0 bytes. A read error after the first chunk ends stdin early with no report.
  • Cost for a window: 3 syscalls at parse, then 4 per 64 KiB, one pread on the JS thread per writable event. Other callers: 0 added syscalls.
Notes

Questions for a maintainer.

  1. Transport. The buffer of spawn: respect Blob offset/size when a sliced Bun.file is used as stdin #31931 is small and reports every read error from spawn(), and on Linux the child gets a memfd. Its cost grows with the window. Honor Bun.file().slice() bounds in Bun.write #41209 carried a port of that buffer until 8206895, and a pre-merge check of that head measured +537 MB of RSS and 616 ms on the JS thread for 512 MiB. A 3 GiB window in a 2 GB container killed the parent. This PR holds one chunk (+2.1 MB for 64 MiB and for 512 MiB, spawnSync +3.6 MB), and it pays with the costs in Downsides. A design review ranked six designs: this one first, the buffer last. A later self-review said that a person must choose between the two.
  2. landing-prs.md:31 forbids a synchronous filesystem call on the JS thread in an async completion path. The refill is one pread of at most 64 KiB in the writable callback.
  3. REVIEW.md:66 asks for agreement before a feature PR adds a shared abstraction. This PR adds Stdio::FileWindow, Source::FileWindow and Source::refill() to the writer that Bun.spawn, the shell and bun install share.
  4. A read error after the first chunk. The writer has no channel to JS, so the error takes the path of a failed write: a debug log, then the child gets EOF.

Self-review. Addressed: the open no longer creates a missing file. A window at stdio[3] throws, and the test no longer pins the whole file there. A groundwork commit that kept an unsafe move is gone, because #40204 rewrites the same arms. This change adds no unsafe. The exclusions are named below. Open: the four questions above, and the merge order.

Merge order. #41209 first. slice(-5) of a file that nothing has stat'd gives an empty stdin until #41257 merges (main: the whole file).

Not changed.

  • The shell form with a path (cat < ${Bun.file(path).slice(1, 3)}): shell_body.rs puts the path text into the command, so the window is gone before Stdio exists. Shell builtins are the same.
  • Bun.stdin.slice() at stdin: the slot inherits.
  • stdout and stderr: the child writes to the file, and a window is ignored there.
  • A file that is not regular (a FIFO, a socket, a terminal, a character device) has no window. The child gets the file, as on main. The parent does not open it.

Measurements (Linux x64, debug build).

  • Parent peak RSS (VmHWM) at this head: spawn +2,056 KiB for a 64 MiB window and +2,104 KiB for 512 MiB. spawnSync +3,660 and +3,600 KiB.
  • Syscalls, from a ptrace logger on an earlier revision of this change (the later edits changed error paths only): a 64 MiB window is 1,024 pread64 of 65,536 bytes and 1,024 sendto, with at most 1 pread64 between two epoll waits. At parse a path window is newfstatat, openat, pread64. A descriptor window is fstat, fcntl(F_DUPFD_CLOEXEC), pread64. Both are the same at 1 MiB and at 512 MiB. An unsliced file adds none.
  • Nine stdin kinds that do not slice ("ignore", "pipe", "inherit", Bun.file(path), Bun.file(fd), a Blob, a TypedArray, the stream of an unsliced file, a JS ReadableStream), 20 spawns each: the same counts of openat, fstat, fcntl, socketpair, memfd_create, sendto and vfork as the base.
  • size_of is unchanged: Stdio 384, Source 24, StaticPipeWriter<Subprocess> 384.
  • Binary size was not measured: no release build of this head exists yet.

Test runs (Linux x64, debug build with ASAN).

  • The new block: 43 pass. On the base (Honor Bun.file().slice() bounds in Bun.write #41209) 32 of the 43 fail.
  • All of spawn.test.ts: one timeout in gcTick > spawn > stdout can be read, which takes 3 to 4.5 s of its 5 s here and passes alone. The machine had a load average above 240 on 16 cores.
  • spawnSync.test.ts 17 pass, spawn-empty-arrayBufferOrBlob.test.ts 3 pass, spawn-stdin-pipe-fd-leak.test.ts 4 pass, bunshell.test.ts 436 pass.
  • bun run rust:check-all: 12 targets pass.
  • CI at 448a2f8 (Buildkite 121866): the new block passed on macOS. On Windows it passed except two cases that compare the path of the error, because the error carried the NT form of the path (\\?\C:\...). ec3c407 reports the path that the caller gave. So the Windows writer and the restore of the file position ran in CI. I did not run Windows or macOS by hand.

Overlap. #40204 (remove unsafe from subprocess.rs and shell/subproc.rs) and #42819 (remove libuv on Windows) rewrite the files of the writer. This PR adds separate match arms and leaves the existing arms as they are.


no test proof · iteration 1 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/spawn/spawn.test.ts

A sliced Bun.file() at stdin gave the child the whole file: the child
got the path or the descriptor of the store, and the window of the Blob
was lost.

Stdio::extract_blob now opens a window of a regular file in the parent,
before a child exists, and reads its first chunk. Stdio::FileWindow
carries it to the StaticPipeWriter as Source::FileWindow. When the
writer has sent a chunk, it reads the next one (64 KiB at most) and
waits for the next writable event. The parent holds one chunk for any
length of the window.

A window that the parent cannot open throws from spawn(). A window at
stdio[3] or above throws: only stdin has a writer in the parent. A file
that is not regular has no window, and the child gets that file as
before.

Co-authored-by: Dylan Conway <dylan.conway567@gmail.com>
@robobun

robobun commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status at ec3c407, two commits on top of #41209 (d36bbf6). Draft: the transport needs a maintainer's choice. The description has the four questions.

Reproduced on 1.4.3 (canary 367d939) with a 26-byte file a..z:

const proc = Bun.spawn({ cmd: ["cat"], stdin: Bun.file(p).slice(10, 20), stdout: "pipe" });
console.log(await proc.stdout.text()); // abcdefghijklmnopqrstuvwxyz

With this PR the child gets klmnopqrst. Bun.spawnSync, new Response(slice), slice.stream() and the shell form < ${Bun.file(fd).slice(10, 20)} give the same result.

The new block in test/js/bun/spawn/spawn.test.ts holds 43 cases. On the base 32 of them fail. With this change all 43 pass on Linux x64. CI ran the block on macOS and Windows at 448a2f8: it passed there except two Windows cases that compare the path of an error, and ec3c407 changes that path.

… opened

On Windows the path that the open gets is the NT form of the path, so
the error carried a path with the \\?\ prefix.
@robobun

robobun commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author
Updated 6:01 AM PT - Sep 30th, 2026

✅ @robobun, your commit ec3c407fc600eca1a5de8bf90ed6e663d66ce3ae passed in Build #121878! 🎉


🧪   To try this PR locally:

bunx bun-pr 44301

That installs a local version of the PR into your bun-44301 executable, so you can run:

bun-44301 --bun

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant