One-line File.read->File.size change wedges the sp_net fiber server under browser load (stuck in poll, server alive but deaf)

[rubys]

Repro: it works as shipped; one line breaks it

curl -fsSL https://rubys.github.io/roundhouse/browse/spinel.tgz -o spinel.tgz
tar xzf spinel.tgz && cd spinel
make build SPINEL=/path/to/spinel        # spinel main.rb --rbs sig -o build/blog
( cd e2e && npm install && npx playwright install chromium && CI=1 npx playwright test )
# -> 6/6 pass in ~2s

Now change the one line in runtime/tep/net.rb (Sock.sphttp_filesize, used to serve static assets):

-    File.read(path).length
+    File.size(path)

Rebuild and rerun the e2e -> 6/6 fail: every page.goto times out at 30s. Same spinel binary, same app, only this line changed.

What "fail" looks like

The server stays alive but deaf — it stops accepting new connections. sample of the wedged process is 100% in the scheduler's poll loop:

sp_Scheduled_s_run_worker  ->  sp_net_poll_run  ->  poll()      [single thread, all samples]

It's blocked in poll() and never wakes for an incoming connection — as if the listen socket dropped out of the poll set.

We tried to reduce it to a bare sp_net + File.size program and couldn't

Everything simpler stays healthy or is build-identical:

• File.size in isolation returns the correct value.
• ~400 concurrent File.size static GETs — healthy.
• parked /cable WebSocket + keep-alive File.size bursts — healthy.
• a comment POST that broadcasts to parked WS subscribers — healthy, and identical on both builds.

Notably, in several of those tests File.size is never called, yet the File.read and File.size builds behave the same — so the File.size codegen change (whole-program inference / layout shift) seems to surface a latent UB in the sp_net scheduler, rather than File.size itself being wrong. Only the full browser workload (File.size-served assets + concurrent /cable WS + a broadcast + multi-worker timing, together) trips it, which is why we couldn't shrink it to a deterministic minimal case.

Build / env

spinel master @ b2e92ea; reproduces on Linux x86_64 and macOS arm64. Roundhouse pinned back to File.read to keep the e2e green (roundhouse@daccd53).

[matz]

Dug into this with the roundhouse tarball. A few things that narrow it down:

The two binaries are semantically identical except one expression. I compiled main.rb --rbs sig both ways and diffed the generated C. The only real change is sphttp_filesize's body:

- return sp_str_length(sp_file_read(lv_path));   // File.read: allocates the whole file as a String, then lengths it
+ return sp_file_size(lv_path);                  // File.size: a stat(), no allocation

Every other diff line (288 of them) is cosmetic block-param temp renumbering (lv_k__bp9770 → …9769, a cascade from one fewer temp). So this is not a codegen-correctness divergence — both builds are the same program. The wedge is purely the runtime effect of dropping the per-asset whole-file allocation, which confirms your "surfaces a latent UB" read.

Ruled out the obvious scheduler suspect (fiber rooting). FiberSlot#@f lowers to a per-instance iv_f and sp_FiberSlot_scan marks it, and the chain APP (const, rooted) → @sched_fibers (PtrArray) → FiberSlot → iv_f (Fiber) is fully GC-marked. So a parked scheduler fiber (incl. the accept fiber) isn't collectible through a rooting gap there.

Couldn't reproduce from asset load alone. 300 concurrent static GETs through sphttp_filesize→File.size, WORKERS=2, even under SPINEL_GC_STRESS=1 — the server stayed healthy (200 in ~0.4 ms after the burst). Notably SPINEL_GC_STRESS=1 did not wedge it, which argues against a "GC sweeps a live object" bug (that gets worse under stress, not hidden) and toward a timing/ordering or uninitialized-state issue: File.size makes asset serving fast (stat vs open+read+alloc), changing the fiber interleaving, and it only trips with the combined /cable WS + broadcast + assets workload.

The diagnostic that would crack it, from your wedged process (you have the browser repro): inspect the scheduler at the wedge. Specifically — is the accept FiberSlot (slot 0) still .alive?, and is its sched_io_fd still the listen fd? That distinguishes two very different failures:

1. the accept fiber terminated (e.g. accept_loop broke out — Sock.sphttp_shutdown_requested read truthy from corrupted/stale memory under the shifted heap layout) → it's gone from sched_fibers, so the listen fd is never re-added to the poll set; vs
2. the poll set still contains the listen fd but poll() never reports it readable / the woken accept fiber isn't resumed.

A gdb backtrace + p sp_net_poll_set[0..sp_net_poll_n] (is the listen fd present?) and the Tep::APP.sched_fibers[0].f state at wedge time would tell us which, and I can take it from there. If (1), the bug is whatever stale read flips the shutdown/term flag; if (2), it's in the poll rebuild/ready accounting.

[rubys]

Got the wedged-process diagnostic (debug build = spinel main.rb --rbs sig --debug of the same tarball + File.size, driven to wedge by the Playwright e2e, lldb on macOS arm64).

It's your case (2), and specifically the fiber list is truncated:

• sp_net_term_flag = 0 and the accept fiber is alive — rules out case (1) (the accept loop didn't break out).
• At the wedge, sp_net_poll_n = 1, active poll fds = [13] (the listen fd, still present).
• Breakpoint in Scheduler.poll_round (scheduler.rb:141): lv_n = 1, lv_added = 1. So Tep::APP.sched_fibers.length == 1 — only the accept fiber remains; every keep-alive / /cable connection FiberSlot is gone.

Baseline on the healthy idle server is identical for the listen state (poll_n=1, fds=[13], term=0), so the difference at the wedge is purely the connection fibers vanishing.

The only code that removes from sched_fibers is the tail-reap in tick (scheduler.rb:81-88):

i = Tep::APP.sched_fibers.length - 1
while i >= 0 && !Tep::APP.sched_fibers[i].f.alive?
  Tep::APP.sched_fibers.delete_at(i)
  # ... parallel arrays delete_at(i) ...
  i -= 1
end

It stops at the first alive fiber, and slot 0 (accept) is always alive — so a pass where the live, parked connection fibers read !alive? truncates the array down to exactly [accept], which is the steady state I observe.

So the suspect is Fiber#alive? (the FiberSlot → Fiber → state read) returning false for still-live, parked connection fibers under the File.size-shifted heap layout, getting them reaped — consistent with the "latent UB surfaced" read. The File.size path drops the per-asset whole-file allocation, which shifts the heap, and the parked-fiber liveness read flips.

Happy to catch the reap in the act (breakpoint the delete_at and dump the victim fiber's state right before it's removed) to narrow alive?-misreport vs actual-early-termination, if that helps.

[rubys]

Caught the reap victims live (breakpoint on sp_Fiber_alive, stop when the checked fiber has state==3, read its raised/raised_cls at the corrected offsets — sp_RbVal is 16 bytes, so raised@96, raised_cls@104).

Every reaped fiber has raised = 0, raised_cls = NULL — they terminated cleanly (the trampoline's normal f->body(f) return, not the setjmp-caught exception path). So:

• Not a crash — no exception escaped the connection handler.
• Not an alive? misreport — state genuinely is 3 (the body returned).

So the connection fibers are returning from their handler loop as if the connection closed, then getting tail-reaped (scheduler.rb:81-88), which cascades down to just the accept fiber (sched_fibers.length == 1 — the wedge in the comment above). That points at the per-connection read loop ending early — recv / sp_net_recv_some reporting EOF / 0 bytes on a still-live socket under the File.size-shifted layout — rather than the GC or the fiber-state machinery.

Happy to instrument the recv loop / dump the sp_net_recv_some return values next if that's the right thread to pull.

[matz]

Following your reap finding — I verified the data path so you don't have to re-check it, and it points at one specific spot:

The chunk faithfully reflects recv. sphttp_recv_some is :binstr, and its lowering is:

lv_chunk = ({ const char *_t641 = sp_net_recv_some(client, 4096);
              sp_str_from_bytes(_t641, (size_t)(sp_net_bin_len < 0 ? 0 : sp_net_bin_len)); });

sp_str_from_bytes copies sp_net_bin_len bytes immediately, in the same statement-expr (no fiber yield between the recv and the length read), so there's no static-sp_net_recv_buf aliasing race and no sp_net_bin_len cross-fiber window here. And sp_file_read/sp_file_size don't touch sp_net_bin_len (only sp_net.c writes it). So the File.size difference is purely timing, and chunk.length == 0 ⇒ sp_net_recv_some genuinely set sp_net_bin_len = 0.

Where that 0 comes from — the EAGAIN/EOF conflation. sp_net_recv_some (lib/sp_net.c:239) returns the same empty result (bin_len = 0, buf[0] = '\0') for both a real EOF (recv → n == 0) and a would-block (n < 0, errno == EAGAIN/EWOULDBLOCK):

n = recv(fd, sp_net_recv_buf, maxlen, 0);
if (n >= 0) break;
if (errno == EINTR) continue;
sp_net_bin_len = 0; sp_net_recv_buf[0] = '\0'; return sp_net_recv_buf;   // EAGAIN lands here too

And read_request_blob treats chunk.length == 0 as a hard close (return ""). So a spurious-readable → EAGAIN (poll reports the fd readable, but the recv then would-block) closes a live connection exactly the way you observed: the handler returns "", handle_one → false, the fiber terminates cleanly (state==3, raised==0), gets tail-reaped, cascade to sched_fibers == [accept]. The faster File.size asset path tightens the accept/recv interleaving, which makes the spurious-readable more frequent — consistent with "only the full File.size workload trips it."

The one value that settles it at the reap: sp_net_recv_some's n and errno (and the fd):

• n < 0 && errno == EAGAIN/EWOULDBLOCK → it's the spurious-readable-mis-closed-as-EOF path. Fix is spinel-side: sp_net_recv_some must distinguish would-block from EOF (e.g. set sp_net_bin_len = -1 on EAGAIN), and the recv loop must re-io_wait rather than close on it. I can land the sp_net half once confirmed.
• n == 0 with fd == the expected client fd → a real FIN on a socket the browser didn't close (points back at the poll/slots ready-accounting waking the wrong fiber, or a double-close of that fd elsewhere).
• fd != the expected client fd → the fiber's captured client got corrupted (layout-sensitive), a different bug.

If you can dump those three at the breakpoint you already have, I'll take the matching half.

[matz]

Landed the spinel-side EAGAIN half on master (1f8cb08), since correct EAGAIN handling is needed regardless of which of the three the dump shows.

sp_net_recv_some / recv_all / write_str / write_bytes now retry on EAGAIN/EWOULDBLOCK as well as EINTR. Instead of busy-spinning the non-blocking fd, they wait for readiness with a single-fd poll (POLLIN for a read, POLLOUT for a write) and retry; the 1s poll timeout rechecks the term flag so shutdown still breaks the wait. So a would-block after a ready poll no longer returns an empty buffer the caller reads as EOF — which is exactly case (1)/EAGAIN from the previous comment.

If the wedge was the spurious-readable→EAGAIN→mis-close path, this should fix it directly — worth a rerun of the Playwright e2e on the File.size build against this commit. If it still wedges, then the dump's n/errno/fd will point at case (2) (real FIN — poll/slots waking the wrong fiber, or a double-close) or case (3) (corrupted client), and I'll take that half.

Verified make test (975, sp_net suite incl.) and that the tep blog fiber server rebuilds + serves cleanly under a concurrent burst with the change.

[rubys]

Closing as a user error. HTTP content length needs to match bytes sent; if content length is greater than bytes sent, the client will wait. Sorry for the noise.