Repository navigation
Conversation
… (Linux) The shell writes `$` output to dups of fd 1 and fd 2. For a poll that its owner did not label, the pipe writer waited for POLLOUT and then called write(2) with the whole buffer. POLLOUT on a Linux pipe proves one free slot, and a slot holds one page. write(2) of more than that on a description without O_NONBLOCK returns only when every byte is in, so the event loop thread slept until the reader drained the pipe. The shell cannot set O_NONBLOCK on that description. Children inherit it, and a plain tool fails with EAGAIN on it. pwritev2(RWF_NOWAIT) covers a pipe(2) pipe from Linux 6.4, a socket from 6.5, and a FIFO never. Once RWF_NOWAIT is refused, try_write looks at the fd on its first write of more than one page (Linux and Android): - A pipe or FIFO: poll, then write(2) of at most one slot, while the description has no O_NONBLOCK. #43868 had this bound in bb0e24a and removed it in b2ca90a, when process.stdout went back to O_NONBLOCK. On macOS the byte count from kqueue already bounds this write. - A socket that its owner labelled as a pipe: it becomes a socket, and send(MSG_DONTWAIT) writes it. - A tty: unchanged. IOWriter::bump moved every unwritten byte to the front each time 128 KB were written. That is quadratic in the queue wherever writes do not wait, on every platform. It now waits until the written prefix is at least half of the buffer.
|
Updated 6:49 AM PT - Oct 21st, 2026
❌ @robobun, your commit f4d814f has some failures in 🧪 To try this PR locally: bunx bun-pr 46307That installs a local version of the PR into your bun-46307 --bun |
|
Status Reproduced on Linux 6.1.155 with release builds of main (71bc818) and of this branch.
PR: #46307 |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
WalkthroughPOSIX pipe writers now track per-descriptor write strategies and select write paths based on descriptor type and flags. Shell output buffer draining also requires the written prefix to cover at least half the buffer. New tests cover FIFO, socket, and TTY output. ChangesShell Output Handling
Suggested reviewers: Priority: ➖ Normal Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation |
|---|---|---|
| Title check |  |
The title clearly summarizes the primary Linux change: limiting blocking pipe and FIFO writes to one slot per POLLOUT for stdout and stderr. |
| Description check | |
The description is complete and relevant. It explains the problem, fix, trade-offs, limitations, and verification results, including detailed test coverage. The Problem/Fix and Tests sections provide … |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
Full details: Description check
-
Explanation
The description is complete and relevant. It explains the problem, fix, trade-offs, limitations, and verification results, including detailed test coverage. The Problem/Fix and Tests sections provide the information requested by the template, although the exact template headings are not used.
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🟣
src/io/PipeWriter.rs— Shell callers whose reader closes mid-chunk now getwritten: 0and lose the capture of pages that were already delivered, on every chunk over one page rather than only on a signal-interrupted write. src/io/PipeWriter.rs:132-143 loops write_one_slot_to_pipe once per page; after N pages succeed, the next page's poll reports Hup and sys::write returns EPIPE, and the Err at src/io/PipeWriter.rs:143 discardsoffset. The contract at src/io/PipeWriter.rs:215-218 ('an error here means nothing was written this round') is no longer true for the Pipe arm. Fix: report the pages already written before the error (e.g. return Pending(offset) and surface the error on the next round) so on_write tees them and parents see the true count.Why this was flagged
Trigger: a non-quiet
$writes a chunk larger than bun_core::page_size() to a blocking pipe or FIFO, and the reader exits after consuming part of it. Pages 1..N succeed andoffsetadvances at src/io/PipeWriter.rs:147; page N+1 polls Hup, sys::write gets EPIPE, and line 143 returns WriteResult::Err(err) with offset dropped. on_poll then calls on_error (src/io/PipeWriter.rs:207) without any on_write, so IOWriter::on_error (src/runtime/shell/IOWriter.rs:862) reportswritten: 0for every pending writer and the bytes already in the pipe are never teed into the captured stdout buffer. On the base this needed a partial blocking write(2) (reader close mid-copy); write_to_blocking_pipe wrote the whole chunk in one call, so in the common case either all bytes were counted or none were written. The PR makes the multi-call loop the normal path for every chunk over one page, so the lost count is now the ordinary EPIPE outcome. The doc comment at src/io/PipeWriter.rs:215-218 still asserts the old invariant.Verification: pre-existing. Trigger: a non-quiet
$whose stdout/stderr fd is a blocking pipe/FIFO and whose reader closes after some of a chunk has been written.try_write_with_write_fn(src/io/PipeWriter.rs:132-143, untouched by the diff) advancesoffseton eachOk(wrote)and thenreturn WriteResult::Err(err)withoutoffset. The base already fails the same way by the same route.
| if state.get() == PipeWrite::OneSlot | ||
| && !sys::get_fcntl_flags(fd).is_ok_and(|flags| (flags as i32 & sys::O::NONBLOCK) != 0) | ||
| { | ||
| return write_one_slot_to_pipe; |
There was a problem hiding this comment.
🟡 (optional) Users relaying a large external command's output through a non-quiet $ to a slow FIFO, socket or pre-6.4 pipe can now see Bun's memory grow to the whole output size. Once the one-slot write at src/io/PipeWriter.rs:311 stops blocking, nothing pauses the child reader, so every unwritten byte accumulates in IOWriter.buf (src/runtime/shell/IOWriter.rs:639 only compacts, never caps). On the base branch the blocked write(2) also stopped the reader, so the child was paced by its 64 KiB pipe. Fix: add backpressure in the shell writer path, e.g. stop watching the subprocess stdout/stderr read poll while the queue of unwritten bytes exceeds a bound and resume when it drains.
Why this was flagged
A script runs await $\cat 4GB.log`(not quiet) with fd 1 a FIFO, a socketpair from a parentBun.spawn`, or a pipe on a kernel without FMODE_NOWAIT, and the reader drains slowly. try_write (src/io/PipeWriter.rs:86-93) now returns write_one_slot_to_pipe (src/io/PipeWriter.rs:311) or write_to_socket (src/io/PipeWriter.rs:302), which return Pending instead of sleeping in write(2). The JS thread keeps running, so the shell's subprocess output reader keeps reading the child's pipe and enqueuing into IOWriter; bump (src/runtime/shell/IOWriter.rs:635-647) only drains the written prefix and there is no cap or pause on the reader anywhere in src/runtime/shell. RSS grows to the size of the command's output until the reader catches up, and a container memory limit kills the process. On the base branch write_to_blocking_pipe (src/io/PipeWriter.rs:257) blocked the thread, the reader did not run, the child blocked on its 64 KiB pipe, and memory stayed bounded.
Verification: nit — acknowledged in diff: the PR description's "Downsides" states "The shell no longer paces an external command. 200 MB to a reader that stalls 3 s peaks at 777 MB RSS (main: 470 MB)". src/runtime/shell/subproc.rs:1852-1876 on_read_chunk never pauses on IOWriter backpressure. src/runtime/shell/IOWriter.rs:1011 does s.buf.extend_from_slice(buf) with no cap.
| } else if s.total_bytes_written >= SHRINK_THRESHOLD { | ||
| } else if s.total_bytes_written >= SHRINK_THRESHOLD | ||
| // The drain moves every unwritten byte. A queue that grows faster than the fd takes it would pay that per chunk. | ||
| && s.total_bytes_written >= s.buf.len() / 2 |
There was a problem hiding this comment.
🟡 (optional) Users relaying a large command's output through a non-quiet $ to a slow reader now hold up to twice the unwritten bytes in the shell queue, on top of the growth already reported. src/runtime/shell/IOWriter.rs:635-637 only compacts buf once the written prefix is at least half of buf.len(), so with a reader that drains slowly the already-written prefix is retained until it equals the unwritten tail. On the base the prefix was dropped every 128 KiB of progress. Fix: bound retained written bytes independently of the queue size (e.g. a ring or VecDeque so compaction is O(1), or compact when the prefix exceeds max(SHRINK_THRESHOLD, unwritten/k) with a hard cap).
Why this was flagged
Trigger: a non-quiet $ relays an external command whose output outruns a blocking FIFO, socket or pre-6.4 pipe reader, through subproc.rs PipeReader chunks enqueued into IOWriter and drained by bump (src/runtime/shell/IOWriter.rs:611). The old condition at the base's IOWriter.rs:635 compacted whenever total_bytes_written >= SHRINK_THRESHOLD (128 KiB), keeping buf at roughly unwritten + 128 KiB. The new clause at src/runtime/shell/IOWriter.rs:637 waits until total_bytes_written >= buf.len() / 2, so with 300 MB queued and 100 MB written nothing is freed; buf can reach 2x the unwritten tail. The confirmed finding at src/io/PipeWriter.rs:311 covers the loss of backpressure; its remedy (pacing or capping the relay) does not change this compaction rule, which doubles whatever is retained for any multi-chunk queue, so the fix and the site are distinct. The PR's own 777 MB vs 470 MB RSS measurement includes this factor but the dismissal treated it as 'only the constant'. Population: every slow-reader relay on Linux FIFOs, sockets on pre-6.5 kernels and pipes on pre-6.4 kernels.
Verification: Triggered when a non-quiet $ relays external-command output into an IOWriter whose fd drains slower than chunks are enqueued. At src/runtime/shell/IOWriter.rs:635-643 the written prefix W is only dropped once W >= U, where U is the unwritten tail, so buf.len() reaches up to 2U, whereas the base condition dropped the prefix at every bump once W >= 128 KiB, holding buf.len() at roughly U + 128 KiB.
Problem
$that prints more than a pipe holds sleeps inwrite(2)until the reader drains. This blocks the JS thread. 200 KB into a FIFO read after 2 s: a 100 ms timer fires at 1952 ms.write_to_blocking_pipe(src/io/PipeWriter.rs:252) polls, then writes the whole buffer. POLLOUT proves one free pipe slot.pwritev2(RWF_NOWAIT)(process.stdout/stderr: clear O_NONBLOCK for spawned children, don't drop console output on EAGAIN #43868) does not cover a FIFO or a pre-6.4 pipe.Fix
send(MSG_DONTWAIT). A tty is unchanged.IOWriter::bumpcompacts its queue once half is written. The old rule is quadratic where writes do not wait.test/js/bun/shell/file-io.test.ts, plus 54 related files. Self-reviewed: 64 concerns raised, 53 addressed (Notes).Background
O_NONBLOCKthere breaks children:head -c 1000000dies with EAGAIN.splice(2)relay and a capacity write when FIONREAD is 0. Both keep full speed. process.stdout/stderr: clear O_NONBLOCK for spawned children, don't drop console output on EAGAIN #43868 declined such machinery.Downsides
cat: 1233 ms (main: 649 ms). Another process's output can split a chunk over one page.console.log,Bun.write(Bun.stdout), FreeBSD, and a one-page write that loses a race.Notes
No user report. This was found by reading
write_to_blocking_pipeduring work on #45565 (closed, superseded by #46021). The tracker has no issue for it. #23636 fixed the same symptom in the shell on macOS. In the #43868 review the shell was left alone after a macOS trace of a socket stdout showed each write capped by the kqueue count (discussion). epoll giveson_pollno count (src/io/posix_event_loop.rs:365), so on Linux the cap isusize::MAX.History of the bound. #43868 first sized this write from
F_GETPIPE_SZandFIONREAD(62a5dfd). bb0e24a replaced that with one page after POLLOUT ("Only POLLOUT's guarantee is safe: one free page slot on Linux"), because queued bytes undercount used slots. b2ca90a then dropped the bound ("Drop the per-call nonblocking write machinery") whenprocess.stdoutwent back toO_NONBLOCK, and 2c5f635 kept the per-call writes that cost nothing:pwritev2(RWF_NOWAIT)andsend(MSG_DONTWAIT). The shell's writers got neitherO_NONBLOCKnor the bound. This PR restores the bound for them, with onefstatper writer where bb0e24a had one per write. The socket arm is thesendthat 2c5f635 already uses.Who is affected. The stall needs a shell output chunk larger than the free room of fd 1 or fd 2, a description without
O_NONBLOCK, and:pipe(2)pipe on a kernel withoutFMODE_NOWAITfor pipes (upstream before 6.4), or withBUN_FEATURE_FLAG_DISABLE_RWF_NONBLOCK,child_processor ofBun.spawnwithstdout: "pipe") on a kernel before 6.5,fs/pipe.c,fifo_open) the same holds for a pipe opened by path (/dev/stdout,/proc/self/fd/N, bash> >(cmd)). Not run on a 6.4+ kernel.Checked here on 6.1.155:
pwritev2(RWF_NOWAIT)returns ENOTSUP for a pipe, a FIFO and a socketpair.It covers builtin output, shell messages on stderr, the output of external commands that a non-quiet
$relays, and builtins underbun execand.shfiles. It does not happen with.quiet()or.text(), for a redirect to a FIFO path, or after the script readprocess.stdoutor calledBun.stdout.writer(). Those setO_NONBLOCKon the description until the next spawn with inherited stdio clears it (checked with/proc/self/fdinfo/1).Repro (Linux 6.1.155, release builds of the merge base 71bc818 and of this branch):
/proc/PID/syscallduring the stallwrite(2)epoll_pwait2bun big.sh(mini event loop)write(2)epoll_pwait2Tests. Six new tests in the
output that its reader does not drainblock.a blocking FIFO on stdoutandon stderr(Linux): the fixture is the only reader of the FIFO and reads it on the thread that runs the shell. It checks a write of one slot, of one slot plus one byte and of the capacity plus one byte while exactly one slot is free. Then a small chunk before a chunk larger than the pipe, two shells at once, and 512 KiB from an external command (this reaches the changed branch ofbump, checked with a breakpoint). It checks that the description is still blocking. Last, it makes the description nonblocking and checks that the shell then needs fewerwritecalls than the pipe has slots (syscwin/proc/thread-self/io).two blocking FIFOs under a shell script(Linux):bun two.shrunsseq 1 30000 1>&2 | seq 1 30000on the mini event loop. stdout must arrive while nothing reads stderr.a socketanda socket without RWF_NOWAIT: a child whose stdout is a socketpair prints two small chunks and 4 MiB. It must report 100 event loop turns while that is pending. The second test setsBUN_FEATURE_FLAG_DISABLE_RWF_NONBLOCKso kernels from 6.5 reach thesendarm.a tty: 100 KiB ofechooutput arrives once and in order. It passes with and without the fix and guards the unchanged tty path.a socketpasses there without the fix.file-io,shell-blocking-pipe,exec,epipe,shell-pipe-read-faultandpipeline_stackpass.terminal,spawn,spawn-maxbuf,filesink,process-stdio,bun-write): the only difference is the five new tests. Two failures are on both builds in this container, which has a slow disk:ls > recursive > node_modulestimes out, and 15 to 18 tests ofbun-write.test.jstime out.Costs, measured. Release builds of 71bc818 and this branch, Linux 6.1.155. Syscalls are main-thread counts from a ptrace counter (strace and perf are not installed here), per
await $echo ${p}`` with stdout a FIFO thatcatdrains. A `pipe(2)` pipe gives the same table on this kernel.The same, after the script read
process.stdout(the description hasO_NONBLOCK, main does not stall):fstatis once per writer. Thefcntl(F_GETFL)is once pertry_writecall of more than one page..quiet(): the syscall tables are identical. A tty: one morefstatper writer for a chunk over one page.pipe(2)pipe on Linux 6.4 or later was not measured (no such kernel here). By the code, RWF_NOWAIT returns before any changed line.cat, 11 interleaved runs, median (min to max):O_NONBLOCKO_NONBLOCK$(head -c N /dev/zerointocat), 10 interleaved runs, median (min to max):bumpchange alone: 128 MiB relayed intocatafter the script readprocess.stdout, so no build waits inwrite(2). Main: 5.3 to 6.1 s wall, 3.1 to 3.7 s user CPU. This branch: 1.2 to 1.6 s wall, 0.07 to 0.16 s user CPU (3 runs each).$keeps the whole output forShellOutputin both builds.O_NONBLOCKThe last row shows that main has the same queue wherever its write does not wait.
write(2)..text+1280 bytes (81,904,741 to 81,906,021), file size unchanged.PosixBufferedWriter(32 bytes) andPosixStreamingWriter(64 bytes) keep their size: the one-byte state replaces the one-byterwf_unsupportedflag.Other visible changes.
console.log, a second$, or the stderr writer under2>&1can land inside a shell chunk. Before, the sleepingwrite(2)kept the chunk whole.process.exit()that races a shell write drops what the pipe has not taken. Before, the process could not exit until the reader drained the pipe.ShellOutput.stdoutholds the bytes that reached the pipe (65536 here). Before it held 0. The exit code is the same.Not covered. stdout is a FIFO read after 2 s. The number is when a 100 ms timer fires, main then this branch.
console.log(200 KB): 1897 and 1941 ms. The console writer is synchronous by design (b2ca90a).process.stdout.write(200 KB)after aspawnSyncwith inherited stdio: 1838 and 1935 ms. process.stdout/stderr: clear O_NONBLOCK for spawned children, don't drop console output on EAGAIN #43868 keeps Node's behavior there (discussion).await Bun.write(Bun.stdout, 200 KB): 1845 and 1953 ms. It writes on the JS thread. It shares no code with the shell's writer, so it is not part of this PR.sys_pipe.c,pipe_direct_write), a blocking pipe write of 8192 bytes or more waits until the reader took it, whatever the kevent count says. Not run. There is no FreeBSD test lane.on_pollcuts a blocking write to the kqueue byte count.bumpchange applies on every platform.Alternatives.
O_NONBLOCKon the shell's dup, asprocess.stdoutdoes. The description is shared with every child that inherits it. Measured: with the flag set,head -c 1000000 /dev/zeroon that description fails withResource temporarily unavailableafter 65536 bytes, exit 1. A spawn clears the flag for the child, so the shell would have to set it again for each write while that child still runs.is_pollable()in the shell already callsfstaton the dup.IOWriter::__startcould label the poll from it, undercfg(linux, android), andtry_writewould lose itsfstatand itsset_flag. This PR keeps the test in the writer:Flags.is_socketalso decidesno_sigpipe, and Remove libuv on Windows #42819 rewrites that part of the shell.ioctl(FIONREAD)is 0. In an emulation (a hook that changes the length of the write in a release build of this branch) it restores the speed of main: 60 x 1 MiB intocattakes 632 ms on main, 1097 ms with one slot and 603 ms with it, median of 9. Its costs: two more syscall wrappers, a lost race waits for up to the whole pipe and not one slot, and a faulting write by any writer leaves a slot in use while FIONREAD says 0 (seen here on 6.1). It can follow if the exposure is acceptable.robobun/e6242f5e/pipe-splice-relaycopies the bytes into a private pipe and moves them withsplice(SPLICE_F_NONBLOCK), which waits for neither pipe. It removes the 2x cost and the race. It costs 2 fds per thread, 16 bytes per writer and 8.7 KB of.text.Self-review. Two rounds, 64 concerns, 53 addressed. The first round moved the PR from the relay to this bound, because #43868 chose "where a per-call nonblocking write costs nothing" (discussion). The second round added the
O_NONBLOCKcheck, the mini event loop test and the cost rows above. Not done:Bun.write.Related. #43091 makes pipeline commands write into a blocking
pipe(2). It does not change the root stdout and stderr writers. #42819 removes the compaction that thebumphunk changes, so the two conflict.no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/shell/file-io.test.ts