Repository navigation
chore: sync upstream Bun with paired OpenClaw WebKit - #124
Merged
Merged
Conversation
…ven-sh#43962) ### What A TLS client that calls `end()` before its first handshake step (in the tick of `tls.connect()`, in the next tick, or inside `'connect'`) sent a bare FIN with no ClientHello. It never reported a certificate that it refuses, and against a server that keeps its side open it never exited. Node sends the ClientHello first. This is a regression that has not shipped: oven-sh#42181 made `_final` send the FIN at once. The script of oven-sh#17190 passes on 1.4.2 and fails on `main`. ### How - `us_internal_ssl_on_open` runs the `'connect'` listeners and then sends the ClientHello. `us_internal_ssl_shutdown` now holds the FIN back in between, and `us_internal_ssl_on_open` sends it right after the ClientHello. - Only `tls.connect()` opts in, from a `'connect'` listener that it registers first. That is where Node registers `_start` ([wrap.js#L1795](https://github.com/nodejs/node/blob/v26.3.0/lib/internal/tls/wrap.js#L1795)). `Bun.connect`, `upgradeTLS`, a second `connect()` of the same `TLSSocket` and `new TLSSocket().connect()` never set the bit. ### Evidence - `node-tls-duplex-end-verify.test.ts`: 68 pass on a debug and on a release build. Under `node --test` (v26.7.0), 67 pass and 1 is skipped. - Regression test for oven-sh#17190. ### Not in this PR With the ClientHello sent, an early `end()` behaves like an `end()` one turn after `'connect'` does on `main`. Three differences from Node that `main` has there now show for an early `end()` too. Measured on earlier heads of this branch: - TLS 1.2, `rejectUnauthorized`, untrusted chain: `error UNABLE_TO_VERIFY_LEAF_SIGNATURE`, where Node reports `end`, `error ECONNRESET`. - More than 10 such clients at once in one process: the ones that wait in the low-priority handshake queue report `ECONNRESET` and never read the server's flight. - A TLS 1.2 session that is offered for another server name: `error ERR_TLS_CERT_ALTNAME_INVALID`, where Node offers no session and reports `ECONNRESET`. Also measured on earlier heads, inside the tick in which the FIN is held: - `end()` and then `resetAndDestroy()`: the FIN or the reset is missing, where `main` sends both. Node throws on `resetAndDestroy()` of a TLS socket. - `_handle.write()` after `_handle.shutdown()` returns 0, where `main` returns -1. `end()` over a Duplex, or over a wrapped socket that is still connecting, is still lost, as on `main`. --------- Co-authored-by: Ciro Spaciari MacBook <ciro@anthropic.com> Co-authored-by: Dylan Conway <dylan.conway567@gmail.com>
…swers or throws for a body that cannot be read (oven-sh#44504) ### What does this PR do? Fixes regressions on main from oven-sh#42053, which is in no release. Replaces oven-sh#43916. ```js const f = Bun.file(path); // 5 bytes new Response(f).clone(); // nothing reads a body await Bun.write(path, "123456789"); f.size; // 1.4.2: 9, main: 5 ``` After `clone()` of a body that is a `Bun.file(path)`: | | 1.4.2 | main | this PR | |---|---|---|---| | file grows 5 to 9 bytes, then `f.size` | 9 | 5 | 9 | | same, then `new Response(f).body`, or `.body` of either clone | `123456789` | `12345` | `123456789` | | `f.lastModified` after `utimes`, `f.exists()` after `unlink` | fresh | stale | fresh | | the path is a directory | both bodies reject with `EISDIR` | `clone()` throws | both bodies reject with `EISDIR` | | `Bun.file(fd)` of a directory, or of an fd that is not open | both bodies reject (`EISDIR`, `EBADF`) | `clone()` throws | both bodies reject (`EISDIR`, `EBADF`) | | a procfs file: both bodies and `f.text()` | whole | `""` | whole | The same holds for `Request.clone()`, `new Request(request)`, `new Request(request, init)` and `new Request(url, response)`. **Cause.** `clone()` has to know whether a path names something that yields its bytes once (a pipe, a terminal), because two Blobs over that would compete and the body is teed instead. oven-sh#42053 asks with `resolve_file_stat`, which writes `mode`, `seekable`, `max_size` and `last_modified` on the `Store`. The body shares that `Store` with the `Bun.file()` it was made from, and nothing invalidates those fields, so `clone()` became one more call that freezes them. Separately, its rule was "tee whatever is not a regular file", and a tee over a directory cannot start. **Fix.** `store_yields_bytes_once` (new, `src/runtime/webcore/Blob.rs`) answers for a Blob body, and `Value::clone_with_readable_stream` calls it in place of `store_reads_repeatably`. - It keeps the mode it stats in a field of its own, `mode_seen_by_clone`, which nothing else reads. That is the fix for the stale answers and for procfs. - It does not count a directory as yielding bytes once, by path or by fd, nor an fd that `fstat` fails for. There are no bytes to compete for, so the body is duped like a regular file and each body fails when it is read, as in 1.4.2. A failed `clone()` on main also leaves the original body used. - For an fd it asks with `resolve_file_stat`, which is what the tee of an fd does next (`to_readable_stream` resolves the size). So the `fstat` only moves earlier, and an fd body is cached as on main. - It still stats once per `Store` and still uses a stat that `size` or `.body` cached earlier, so `clone()` makes the same syscalls as on main. **Kept from oven-sh#42053:** a body over an fd, `Bun.stdin`, a FIFO by path and a terminal by path is teed, and a cloned `Bun.file().stream()` body keeps its type. **This is a partial fix.** The arm for an unread `.stream()` body is not touched, so `new Response(f.stream()).clone()` still caches the stat: `f.size` and the rest stay stale after it, and a procfs `.stream()` body reads `""`. Taking the cache away there makes the bodies, which oven-sh#42053 turns back into Blobs, behave in ways that neither 1.4.2 nor main has. Most of those are the right value, but `console.log` of a cloned `slice().stream()` body would print `0.0 ZB`. That needs its own decision, so it is left out. **Nothing that main answers correctly after a `clone()` gets worse.** Main has two answers right only because `clone()` writes the cache. Taking the write away alone would bring two older bugs back for code that clones, so they are fixed where they are: - `f.exists()` stayed `false` for good once a stat had failed, with or without a `clone()` (`await f.exists()`, create the file, `await f.exists()`). `get_exists_sync` now stats again while no stat has succeeded. - `console.log` of a `Response` or `Request` over `Bun.file(p).slice(1, 4)` prints `0.0 ZB` until something stats the file, and on main `clone()` was that something. `get_size_for_bindings` now also accepts the mode `clone()` saw, so a cloned slice prints `3 bytes` as on main. - `0.0 ZB` is `u64::MAX`, "unknown", run through the byte formatter. It is no longer printed: `Response {` and `Request {` carry no size when it is unknown. This is what makes the label right for a file that was deleted after the clone, where main prints the size it had. - Printing such a body left the 0 of the failed stat in it. The next `console.log` would then have printed `0 KB`, and `text()` read `""` even after the file was created, with or without a `clone()`, on 1.4.2 too. `get_size_for_bindings` no longer keeps that 0. A slice that was never cloned still has no size that `console.log` can print. It printed `0.0 ZB` and now prints none. An empty file and a procfs file give the same `stat`, so cutting the slice to `st_size` gets one of them wrong, and I left that alone. **Why not oven-sh#43916.** It drops the stat for a Blob body, so a FIFO or a terminal given by path as the Blob is duped again and one body gets nothing. It stats uncached for a `.stream()` body, which costs 1000 `newfstatat` per 1000 clones of streams from one shared `Bun.file()`, where main pays 1. And it has the new behaviours described under "partial fix". ### How did you verify your code works? Tests, each run on a debug build of main, on this branch and on 1.4.2: | | main | here | 1.4.2 | |---|---|---|---| | `body-clone.test.ts`, 12 regression tests: `size` and a later `.body` through each of the five ways to clone, `lastModified` and `exists()`, both bodies after the file grew, the size `console.log` prints, a procfs file, a directory by path and by fd, an fd that is not open | fail | pass | 11 pass, the printed size fails | | `body-clone.test.ts`, 3 tests that pin what main has right: a FIFO given by path as the Blob itself is teed, so is a regular file given by fd, and `exists()` sees a file that was created before the clone | pass | pass | fail | | `bun-file-exists.test.js`, 2 tests: one `Bun.file()` sees a file that is created after `exists()` or `size` found none | fail | pass | fail | | `inspect.test.js`, 2 tests: no size is printed for a missing file or a directory, and printing a body whose file does not exist yet leaves it as it is | fail | pass | fail | With the `is_fd` clause of `store_yields_bytes_once` disabled, the fd test is the only one in `body-clone.test.ts` that fails. A differential run on Linux x64 of this branch against a debug build of main at the same base. It crosses 12 kinds of body (regular, empty, large, symlink, missing, directory, `/dev/null`, procfs, fd of a file, fd of a directory, fd that is not open, in-memory Blob), 4 earlier reads of the handle, 6 body forms, the 5 ways to clone plus 2 controls that do not clone, up to 12 changes to the file before or after the clone, and 29 observations, each in a fresh directory with a fresh handle. Two of them do a thing twice: print twice, and print and then read. The rule it checks: **a row that differs from main has to be correct**, not merely what an older version did. Each scenario records what is on disk at that moment (existence, size, mtime, content). A changed row has to match that, or, where no such truth can be derived, what the same scenario answers without a `clone()`. A row where main matches the truth and this branch does not is reported as a regression on its own. There is none. | of 187,404 rows | | |---|---| | same as main | 183,155 | | differs, matches what is on disk | 3,108 | | differs, same as without a `clone()` | 775 | | differs, prints no size where main prints `0.0 ZB` and the size could be known | 342 | | differs, wrong | 0 | | a `/dev/null` body served with `Bun.serve`, which flips between a reset and a 500 page on main between its own runs | 2 | | not an observation of `clone()`: `fetch()` refuses a GET `Request` with a body, for every kind of body on both builds | 22 | Extra syscalls for 1000 clones (`strace`, N=1000 minus N=0), `newfstatat` / `fstat` / `openat`: | | before oven-sh#42053 | main | this PR | |---|---|---|---| | `new Response(Bun.file(p))` | 0/0/0 | 1000/0/0 | 1000/0/0 | | one shared `Bun.file()` | 0/0/0 | 1/0/0 | 1/0/0 | | `Bun.file(p).stream()` body | 0/1000/1000 | 1000/0/0 | 1000/0/0 | | `shared.stream()` body | 0/1000/1000 | 1/0/0 | 1/0/0 | | `Bun.file(fd)` body | not measured | 0/2000/1 | 0/2000/1 | | one shared `Bun.file(fd)` | not measured | 0/1001/1 | 0/1001/1 | Unchanged from main: 48 cases over a FIFO, `/dev/stdin`, `Bun.stdin` and a file replaced by a FIFO (3 runs each), `/dev/tty` and `/dev/stdin` under a pty, and `/dev/null` and `/dev/zero` slices. Suites on the debug build, no failures: `body-clone`, `body`, `response`, `blob`, `blob-write`, `body-stream`, `request`, `FormData`, `bun-file`, `bun-file-exists`, `inspect`, `console-log`, `bun-serve-file`, `bun-serve-static`. In `fetch.test.ts` and `bun-write.test.js` one test each fails with `EFBIG`, here and on main, because I ran them under a 200 MB file size limit. `bun run rust:check-all` passes for `x86_64-pc-windows-msvc` and `aarch64-apple-darwin`. Not verified: Windows and macOS at runtime. The two fd tests are skipped on Windows.
…n-sh#44519) Fixes oven-sh#44075. Closes oven-sh#44449. On macOS, dns.lookup(), fetch() and sockets now resolve split-DNS names, for example a VPN resolver behind iCloud Private Relay. getaddrinfo(3) resolves these names, but Bun reported ENOTFOUND. mDNSResponder asks the next resolver after a negative answer only when the client sends the failover attribute. DNSServiceGetAddrInfoEx (oven-sh#40675) accepts that attribute but never sends it, and the daemon reads it only for QueryRecord requests. This PR does what libinfo does: one DNSServiceQueryRecordWithAttribute per family. macOS 12 falls back to plain DNSServiceQueryRecord on the same code path. Thanks to @mimi1vx for the diagnosis. Compared with oven-sh#44449: the hostname is sent as written (a trailing dot turns off search domains, so http://intranet/ would stop resolving), a link-local AAAA keeps its interface scope (fe80::1%7), and removed records are still ignored. Verified: new tests put a scripted mDNSResponder behind the unix socket of the daemon and check the requests and the answers. All 10 pass with bun bd test test/js/bun/dns/. The 9 that existed at that time all fail on 1.4.3 with USE_SYSTEM_BUN=1, and the tenth was not run there. The requests are now identical to those of getaddrinfo(3), and 21 lookups through the real daemon match 1.4.3. Not verified: real failover, which needs a split-DNS machine. @mimi1vx @sassdavid, can you try this build with bunx bun-pr?
…ods (oven-sh#43452) ### Problem - The per-encoding write methods of `Buffer` (`utf8Write`, `latin1Write`, `asciiWrite`, `hexWrite`, `base64Write`, `base64urlWrite`, `ucs2Write`, `utf16leWrite`) accept a value that is not a string. `Buffer.alloc(4).utf8Write(123)` returns `3`, and an object value's `toString()` runs. Node throws `TypeError [ERR_INVALID_ARG_TYPE]: argument must be a string`. - The cause: both host function templates start with `toStringOrNull()` on the value (`src/jsc/bindings/JSBuffer.cpp:2453`, `:2487`). - No user reported this. A comparison with Node found it during work on oven-sh#30138. ### Fix - The new helper `stringArgumentOrThrow` checks `isString()` and throws Node's error. Both templates call it in place of `toStringOrNull()`. - The order follows Node. `utf8Write`, `latin1Write` and `asciiWrite` check the bounds first, so `ERR_BUFFER_OUT_OF_BOUNDS` wins. The others reject the value before they read `offset` or `length`. - `Buffer.prototype.write` is not changed. oven-sh#30138 changes its value check (oven-sh#43451) and adds the identical helper block, so the two PRs merge in either order. - Verified: `test/js/node/buffer.test.js`, block `with a value that is not a string`: 35 tests fail on canary 1.4.3 and pass here. Also the rest of that file and the vendored `test-buffer-*.js` files. ### Background - In Node these methods are C++ (`StringWrite` and `SlowWriteString` in `src/node_buffer.cc`). Both start with `THROW_AND_RETURN_IF_NOT_STRING`: a fixed message, no conversion of the value. - Node wraps `utf8Write`, `latin1Write` and `asciiWrite` in JS that checks the bounds before the C++ code runs. That gives the two orders. - `toStringOrNull()` is JSC's `ToString`. For an object it runs user code, which can detach or resize the buffer. <details><summary>Notes</summary> **Behavior change.** Code that passed a non-string value to these methods got the value's string form written. It now gets `ERR_INVALID_ARG_TYPE`, as in Node. The string path does not change: `toStringOrNull()` returned the same `JSString` for a string. **Probe, Node v26.3.0 against canary 1.4.3 (367d939).** `buf` is `Buffer.alloc(8)`. `A` is `ERR_INVALID_ARG_TYPE` "argument must be a string". `B` is `ERR_BUFFER_OUT_OF_BOUNDS`. | call | Node | canary | this PR | | --- | --- | --- | --- | | `utf8Write(123)` | A | `3` | A | | `utf8Write()` | A | `8` (writes `"undefine"`, the start of `"undefined"`) | A | | `utf8Write(Symbol())` | A | `TypeError`, no code | A | | `utf8Write(new String("ab"))` | A | `2` | A | | `hexWrite(123)` | A | `1` | A | | `base64Write(null)` | A | `3` | A | | `ucs2Write({})` | A | `8` | A | | `utf8Write(spy)`, `hexWrite(spy)` | A, no call into `spy` | `Symbol.toPrimitive` runs | A, no call | | `utf8Write(123, -1)`, `utf8Write(123, 0, 9)` | B | B | B | | `utf8Write(123, 8)`, `utf8Write(123, 0, 0)`, `utf8Write(123, NaN)` | A | `0` | A | | `utf8Write(123, offsetObject, lengthObject)` | A, after both `valueOf` calls | `1` | A, after both `valueOf` calls | | `hexWrite(123, -1)` | A | `ERR_OUT_OF_RANGE` | A | | `hexWrite(123, 9)` | A | B | A | | `hexWrite(123, offsetObject, lengthObject)` | A, no `valueOf` call | `1` | A, no `valueOf` call | | `utf8Write(123)` on an empty or a detached buffer | A | `0` | A | `latin1Write` and `asciiWrite` behave like `utf8Write`. `base64Write`, `base64urlWrite`, `ucs2Write` and `utf16leWrite` behave like `hexWrite`. After the fix, the 293-line probe output is identical to Node's for these methods, except for a receiver that is not a Buffer (see below). **The new tests against Node.** The block uses `bun:test`, like the blocks around it. I ran the same block under Node v26.3.0 with a small shim for `describe`, `it.each` and `expect`. 31 of 31 tests pass there. The other 4 are for `utf16leWrite`, which Node does not have. It shares the host function of `ucs2Write`. Upstream Node has no test that asserts this message for `Buffer`, so there is no upstream file to port. **No user code runs.** The test passes a Proxy whose handler is also a Proxy, so a lookup of any trap is recorded. The canary records two `get` traps and writes 1 byte. Node and this PR record nothing. This also guards against a later switch to `Bun::V::validateString`, which reads `constructor.name` from the value to build its message (oven-sh#43451). **The whole class.** Node uses `THROW_AND_RETURN_IF_NOT_STRING` at four places in `src/node_buffer.cc`. Two are `StringWrite` and `SlowWriteString`, which this PR covers. The other two are `Btoa` and `Atob`. User code cannot reach them with a non-string: `btoa(123)` returns `"MTIz"` in Node and in Bun, because the JS layer converts the value first. **The `write()` sites this PR leaves**, in `jsBufferPrototypeFunction_writeBody`, with line numbers on this branch: - `JSBuffer.cpp:2565`, the `buf.write(value, encoding)` form: a bare `toString()`, so `buf.write(123, "utf8")` returns `3`. - `JSBuffer.cpp:2554` and `JSBuffer.cpp:2596`: `Bun::V::validateString` throws the right code with a different message, and `:2596` runs before `parseEncoding` (`:2605`), so `buf.write(123, 0, "bogus")` throws `ERR_INVALID_ARG_TYPE` where Node throws `ERR_UNKNOWN_ENCODING`. oven-sh#30138 owns all three, and its description links oven-sh#43451. At head `2edc3da077` its `writeBody` calls `stringArgumentOrThrow` at each site, after it resolves the encoding. I read that in its source. I did not build that branch. **Merge checks** with `git merge-tree --write-tree`, this branch at `5cf21732dd`: - Clean with `main` (`519963edc8`), with oven-sh#30138 (`2edc3da077`) and with oven-sh#43368 (`6cd01edf50`). - With oven-sh#30138 in either order, the merged `JSBuffer.cpp` has one definition of `stringArgumentOrThrow` and five call sites. The helper block, with its comment, is byte-identical in the two PRs. - oven-sh#43470 (`8e6c5aa3d9`) conflicts with this branch on the comment above the helper, because it has the earlier three-line comment. It conflicts with oven-sh#30138 on the same lines. **Comments.** The last commit (`5cf21732dd`) shortens three comments and changes no code. The comment above `stringArgumentOrThrow` now has the text that oven-sh#30138 uses, so that git merges the two identical blocks without a conflict. **Deleted guard.** In `jsBufferPrototypeFunction_StringWriteWithEncoding` the detach check was `(strValue.isObject() || offsetValue.isObject()) && isDetached()`. The value is now always a string at that point, so `strValue.isObject()` is gone. **Not changed: a receiver that is not a Buffer.** `Buffer.prototype.hexWrite.call({}, "ab")` throws a `TypeError` with no code (`Expected ArrayBufferView`). Node throws `ERR_INVALID_ARG_TYPE` "argument must be a buffer". The `<enc>Slice` methods have the same difference, so it is a separate change. oven-sh#44525 tracks it. **Not changed: the conversion of `length` runs before the offset check.** With an `offset` that is out of bounds, `utf8Write`, `latin1Write` and `asciiWrite` call the `valueOf` of an object `length` before they throw `ERR_BUFFER_OUT_OF_BOUNDS`. Node throws for the offset first. The order is the same on `main`. A review of this PR found it. oven-sh#44526 tracks it. **Exception scopes.** The new tests pass under `BUN_JSC_validateExceptionChecks=1 BUN_JSC_dumpSimulatedThrows=1` with the debug build. Every one of them takes the throw path. **Suites run with the debug build (ASAN).** On this branch at `5cf21732dd`, which includes `main` at `519963edc8`: - `test/js/node/buffer.test.js`: 718 pass, 1 skip, 0 fail, run with `--timeout 120000`. The file runs its `write()` tests a second time through a JS `write` that calls `utf8Write` and the other methods. - My machine was heavily loaded by other work during that run. With the default 5 s limit, 18 tests of this file failed in one run, all by timeout, and the 35 new tests passed. `main` at the same commit passed 683 of 684 with the default limit, in 78 s, a few hours earlier. - The 35 new tests under `BUN_JSC_validateExceptionChecks=1`: 35 pass. - `test-buffer-write.js` and `test-buffer-write-fast.js` in `test/js/node/test/parallel/`: exit 0. - The 293-line probe: the output differs from Node's only in the receiver lines and the `write()` lines. At the first commit of this PR (`2007e66a6e`), which has the same code before the comment change and the merge of `main`: - `test/js/node/test/parallel/test-buffer-*.js`: 62 of 63 files exit 0 when run directly. The other one, `test-buffer-resizable.js`, uses `node:test` and needs the test runner. It passes there (2 pass). - `buffer-compare-bounds`, `buffer-concat`, `buffer-copy-fill-detach`, `buffer-from-symbol-to-primitive`, `buffer-indexOf-detach`, `buffer-utf16`, `buffer-inspectmaxbytes`, `buffer-resolveObjectURL` (`test/js/node/*.test.ts`): all pass. - `test/js/node/buffer-jit.test.ts`: 11 pass, 1 fail. `differential fuzzer` exceeds its 120 s limit on my machine, a debug build with few cores. It exceeds the limit in the same way with `src/` at `origin/main`, and the file calls none of the methods in this PR. I saw this only locally, not in CI. **Self-review.** The review asked for three things in this description: name the `write()` sites that this PR leaves, name an owner for them, and say that no user reported the bug. All three are done. It also weighed a fix for `buf.write(value, encoding)` in this PR and advised against it, because oven-sh#30138 and oven-sh#43368 both change that line and already conflict with each other on it. The last commit changes comment lines only, so it had no new review. </details>
…Bun.sleepSync (oven-sh#44560) Bumps mimalloc to oven-sh/mimalloc#59. - mi_heap_new + mi_heap_destroy (every MimallocArena) is about 7x cheaper with one allocation and 13x cheaper when unused, and no longer slows down when several threads create heaps at once. - Purging stops doing work that is undone right away: an allocation takes resident memory before purged memory, purge_delay is a minimum age while threads are busy, and a park no longer wakes the scavenger every time. - Memory that other threads free into the pages of a thread that stays blocked is given back after 30 s instead of when that thread wakes up. - The event loop only hands its heaps to the scavenger on a poll that can block. - Bun.sleepSync hands them over as well, so what the thread freed does not stay resident for the whole sleep.
…ory (oven-sh#44564) Bumps WebKit to oven-sh/WebKit#768. Nothing else is in the range. - JSC's worker threads (wasm and JIT compilers, GC helpers) release the free memory of their mimalloc heap after 100 ms idle instead of when they exit after 10 s. Fixes oven-sh#41438. - A thread that blocks in Atomics.wait for over 100 ms does the same. It used to keep everything it had freed for the whole wait. The wasm test is from oven-sh#41449.
Keep the parser format hint separate from the package-scope source tag. Adapts oven-sh#40940 for the async runtime loader.
Bumps mimalloc to oven-sh/mimalloc#60, which merges upstream v3.5.3 into the fork (it was at v3.5.2). - Fixes a crash when the first mimalloc call on a new thread is mi_realloc(NULL, n), mi_reallocf, mi_strdup, mi_mallocn or mi_new, with a TLS model where the thread local heap of a new thread starts out as NULL (macOS and Windows). - Fixes mi_free_size_aligned of a small block with an alignment above its size. Bun's build (MI_FREE_USE_PAGEMAP=1) was not affected. - No function is removed from mimalloc.h, and no signature that src/mimalloc_sys binds changes. Instruction counts for the allocation and free paths are the same before and after.
…for it
Every Dockerfile fetched the key that signs SHASUMS256.txt from
keys.openpgp.org at build time, falling back to keyserver.ubuntu.com.
keys.openpgp.org is resetting connections ("keyserver receive failed: End
of file"), and keyserver.ubuntu.com has never had this key ("No data"), so
every image build fails. That fails the four test-docker-build-*.ts tests
on every linux x64 lane and the "Release to Dockerhub" jobs.
The public key is now written into the Dockerfile, so verifying a release
needs nothing but github.com, which the build already downloads from.
The fingerprint stays pinned: --recv-keys only accepted a key with the
fingerprint it asked for, and --assert-signer now requires the signature
to be made by it, whatever else is in the keyring.
--assert-signer fails when a file named like the signature without ".asc"
exists, which gpg's own --output created, so the verified checksums are
written to SHASUMS256.verified.txt. That also removes the "file
'SHASUMS256.txt' was NOT verified!" warning every build printed.
dirmngr was only there to reach keyservers.
…44507) ### Problem - A TLS 1.2 server can kill a bun process that calls `fetch()` twice. It renegotiates on the idle keep-alive connection, and the next request that takes the pooled socket dies: `panic: abort() called`, stack `SSL_set_session` <- `session_cache::install` <- `HTTPClient::on_open`. Regression in 1.4.0. - `on_open` (`src/http/lib.rs`) runs per request. Its TLS setup was guarded by `SSL_is_init_finished(ssl) == 0`, also true during a renegotiation, so it offered the cached session again. BoringSSL aborts on that. ### Fix - The TLS setup (SNI, ALPN, inline reject, session offer) becomes `configure_tls`. Only `on_connect` calls it, the open callback of a new connection. The guard is gone. - Correct because uSockets starts the handshake after that callback returns, and only a new connection takes that path. - Verified: `test/js/web/fetch/fetch.tls.test.ts` (new test, fails on 1.4.3-canary). Also ran the fetch-keepalive, fetch-session, proxy and tls renegotiation suites. ### Background - fetch parks idle connections in a pool and caches one TLS session per origin (oven-sh#36598). `SSL_set_session` offers it, legal only before a handshake starts. - A renegotiation is a second handshake on a live TLS 1.2 connection, started by the server. - Considered a guard in `session_cache::install` and a check at pool pickup. Both leave per-connection work on the per-request path. ### Downsides - A request that takes a socket mid-renegotiation still stalls until its timeout: uSockets does not retry the parked write (oven-sh#37094). A `todo` test records it. - Cost, release builds: a new TLS connection goes from 1,427 to 1,415 instructions, a pooled request from 5,146 to 5,123. Text grows by 1,024 bytes. The stripped binary keeps its size. <details><summary>Notes</summary> **How to reach it.** No user code is needed beyond two `fetch()` calls to one HTTPS origin with verification on (the default). The server decides the rest. It does not have to withhold anything: a server that answers and then calls `renegotiate()` is enough when the second request starts inside the renegotiation. A server that never finishes the renegotiation keeps the window open for as long as it likes. **Why the guard was wrong.** `SSL_in_init()` is true whenever a handshake object exists and is not finalized. That covers the time before the first handshake step and the whole of a renegotiation. uSockets allows a client renegotiation (`ssl_renegotiate_explicit`) and runs it from its `SSL_read` loop, so the pool's idle data handler never sees the HelloRequest and the socket stays parked while it renegotiates. **What else the old block did to a renegotiating pooled socket.** It set SNI and ALPN again. `session_cache::install` took the cached session out of the cache before it offered it, so the next new connection could not resume. It replaced the armed session sink with an unarmed one. The new test checks the second point: after the pooled pickup, a new connection still resumes the cached session. **The test.** `fetch.tls.renegotiation-peer-fixture.mjs` runs under Node, because BoringSSL cannot send a HelloRequest. It is a TLS 1.2 keep-alive origin behind a TCP relay that can hold the client's bytes, plus a plain HTTP control server. The client fixture parks a connection, asks the origin to renegotiate, and gets the answer only when the relay holds the client's renegotiation ClientHello. Then it starts the request that takes the pooled socket. A control request queued behind it on the HTTP thread proves that the pickup ran. The test uses no timer. **Measured, linux x64.** - 1.4.3-canary.1 (`367d939d9`), release: the new test fails. The client prints `panic: abort() called` at the pooled pickup. - Debug build with ASAN of main `468efacace` without the change: the client exits with SIGABRT at the pooled pickup. - This branch, debug build with ASAN: the new test passes. `fetch.tls.test.ts` (62 pass, 1 todo), `fetch-keepalive.test.ts` (49), `fetch-session.test.ts` (34), `test/js/bun/http/proxy.test.ts` (97) and `test/js/node/tls/renegotiation.test.ts` (21) pass. With `--todo`, the todo test times out, as expected. `fetch-keepalive.test.ts` needed a second run with a longer per-test timeout: four tests hit the 5 s limit on the loaded build machine. - Without a relay (the origin answers, then renegotiates, and the client waits 1, 2 or 3 ms and fetches again), release builds, 5 runs per wait: main `80de08a1db` aborts in 13 of 15 runs. This branch aborts in 0 of 15. **What stays.** In 12 of those 15 runs on this branch the second request does not get an answer. The server reports that the renegotiation ended, and the request ends in a timeout. (Of the other 3, one got its answer and two failed with `EPROTO`, which main also shows in this cell.) The request write parks while the handshake runs, and uSockets dispatches the writable event only in its `HANDSHAKE_COMPLETED` state, which a renegotiation leaves until application data arrives. That is a separate defect in `packages/bun-usockets/src/crypto/openssl.c`, and oven-sh#37094 has a fix for it. It does not come from the session cache: it also occurs with `BUN_FEATURE_FLAG_DISABLE_FETCH_TLS_SESSION_CACHE=1`. The todo test next to the new test awaits that request, so it flips when the write is retried. **Cost numbers.** Release builds of main `80de08a1db` and of this branch's source on the same commit. - Instructions from entry to return, nested calls included, counted with gdb single-steps. 3 runs, 3 consecutive calls per run. - Open callback of a new TLS connection (`Trampolines<uws_handlers::HTTPClient<true>>::on_open`): 1,427 on main, 1,415 here. All 9 calls of each build agree. - `HTTPContext<true>::connect` for a pooled TLS request, sixth request of the process: 5,146, 5,160, 5,146 on main and 5,123, 5,118, 5,123 here. Later requests vary more from run to run: 6,617 to 6,762 on main, 6,585 to 6,723 here. - `session_cache::install` calls: 1 per 1,000 keep-alive requests and 100 per 100 new connections, on both builds. - Size: `bun-profile` text is 80,727,274 bytes on main and 80,728,298 here (+1,024). The stripped `bun` is 80,868,936 bytes on both. Symbols: `HTTPClient::on_open::<true>` 1,474 to 280 bytes, `HTTPContext<true>::connect` 5,283 to 5,798, the open callback 375 to 1,582. The TLS block is now part of the open callback, and the compiler inlines the smaller `on_open` at the three pool reuse sites. - Tools: gdb 16.3, `llvm-nm`, `llvm-size`. `perf`, `valgrind` and `bloaty` are not installed on the machine. **Related.** oven-sh#41671 covers the other way into the same BoringSSL abort, `socket.setSession()` after the handshake started. This change does not depend on it. **Other designs.** - A guard in `session_cache::install`: it stops the abort, but the cached session is still taken out of the cache and the sink is still replaced. - A stricter check in `on_open`: BoringSSL has no query for "the handshake has not started", so it needs a uSockets accessor or new per-socket state, and one branch stays on every pooled request. - Skip a socket that is mid-handshake at pool pickup: the abort stops only because `on_open` is not reached with such a socket. It also changes which requests reuse a connection. - Pass SNI, ALPN and the session to uSockets at connect time: a larger change across every TLS client. It is not needed to remove this defect. **Not run locally.** Windows and macOS. </details>
Carry #106 as the source prerequisite for the rebased OpenClaw WebKit release. Keep import namespaces unmarked and give require(esm) its own live-export facade; preserve forwarded native namespaces and cached require markers. Remove the former prototype/private-field workaround. Preserve the work from ea461c8, fba679a and 72f230b. Adapts upstream Bun oven-sh#33894 and oven-sh#39888 by robobun; original attribution and regression coverage are retained. This prerequisite lands together with the engine manifest and upstream sync.
Merge oven-sh/bun through c7b06d9 while preserving fork resolver, module-loader, stack, worker, VM and idle-accounting changes. Consume immutable OpenClaw WebKit 42ab38d705d4838748ccee77e7deb0e4e35515ee by checksum, paired with the distinct namespace prerequisite from #106. Retain the scalar-only bun: package-import exception and append-only changelog history.
Preserve the original commit from #117 and its contributor credit. Extended namespace qualification exposed the existing async CommonJS package-tag failure on both fork main and the first sync candidate. Node keeps the exports object for a type:module package; keep parser format and package interop metadata separate rather than weakening that assertion.
Import oven-sh#44507 as an additional merge, retaining both histories. Keep the published OpenClaw WebKit pin and fork default-CA regression coverage unchanged.
… and write nothing past a statement being parsed (oven-sh#44235) ### Problem - `Bun.SQL` (postgres): after a pipelined query fails, two queries issued together on that connection resolve with each other's rows. Equal column types: no error. Different column types: `ERR_POSTGRES_UNSUPPORTED_INTEGER_SIZE` or wrong values. - The `ErrorResponse` arm of `PostgresSQLConnection::on` released the failed request before its `ReadyForQuery`. A statement issued in that gap wrote its Parse at once. The late `ReadyForQuery` then made `advance()` write the next request's Bind first. - Behind pgdog 0.1.60, a second `ReadyForQuery` for one Sync does the same on every run. ### Fix - The `ErrorResponse` arm rejects through `reject_in_flight` (`on_undecodable_row` of oven-sh#43187, renamed). The promise rejects at once. The request keeps its counter and the head of the queue until its `ReadyForQuery`. - `advance()` stops at a request whose statement is being parsed, as in oven-sh#20986, and a debug assertion states that rule. A reply carries no request id: it goes to the head of the queue. - Verified: `test/js/sql/postgres-prepared-pipeline-reorder.test.ts` (24 new tests, 19 fail on main), `sql.test.ts`, every `test/js/sql/postgres-*.test.ts`. - Self-reviewed: 7 concerns, 6 addressed, 1 in part (Notes). ### Background - Bun writes several prepared queries before the first reply. The server answers each Sync with one `ReadyForQuery`, also after an error. - `advance()` walks the request queue. It pops finished requests and writes queued ones. - Considered a count of Syncs: two counter updates per query, no result changes. Considered only the clear of `IS_READY_FOR_QUERY` from oven-sh#35120: the step-over in `advance()` stays. ### Downsides - A new statement or simple query issued between an `ErrorResponse` and its `ReadyForQuery` waits for that `ReadyForQuery`. - Release binary: `.text` +256 B. <details><summary>Notes</summary> **Reproduction.** One connection (`max: 1`, also `sql.reserve()` and a pool connection that carries more than one query), a real PostgreSQL, default options: ```js const sql = new SQL({ url, max: 1 }); const ofBob = () => sql`select owner, secret from swap_b where id = ${1}`; const check = n => sql`select 10 / ${n}::int as q`; await ofBob(); await check(1); await Promise.all([ofBob(), check(0)]).catch(() => {}); const [a, b] = await Promise.all([sql`select owner, secret from swap_a where id = ${1}`, ofBob()]); // a is the row of swap_b, b is the row of swap_a ``` The `ErrorResponse` of `check(0)` and its `ReadyForQuery` must arrive in two reads. On loopback that happens in 35 to 86 of 100 rounds. A proxy that forwards every byte and holds the `ReadyForQuery` until the two reads are issued makes it 50 of 50. The tests use that proxy (`pgHoldingProxy` in `test/js/sql/wire-frames.ts`), with no timer. Frames after the rejection, main: `Parse(swap_a) Sync`, the held `ReadyForQuery`, `Bind(swap_b) Sync`, `Bind(swap_a) Sync`. The queue order is swap_a, swap_b. This branch: the held `ReadyForQuery`, `Parse(swap_a) Sync`, `Bind(swap_a) Sync`, `Bind(swap_b) Sync`. The queries in the gap do not have to come from the rejection handler. Issued by a timer that was armed before the failing query, through the same proxy, they are wrong in 10 of 10 rounds on 1.4.2 and in 0 of 10 here. **pgdog.** pgdog 0.1.60 with its query parser on (one primary and one replica entry for the same PostgreSQL 17) rejects a statement with a syntax error itself. It answers the batch `Parse Describe Bind Execute Flush Sync` with `ErrorResponse ReadyForQuery`, and then with one more `ReadyForQuery`. The second one arrives when the Parse of the next request is on the wire. ```js const bad = sql`selec 2 as x`; // rejected by pgdog const a = sql`select 'alice' as owner, ${1}::int as id`; // new to the connection const b = sql`select 'bob' as owner, ${1}::int as id`; // prepared await Promise.allSettled([bad, a, b]); ``` Official 1.4.2: `a` resolves with bob and `b` with alice in 50 of 50 rounds, with no proxy and no mock. This branch: 0 of 50. After the two `ReadyForQuery`, 1.4.2 writes `Parse(alice) Describe Sync`, `Bind(bob)`, and later `Bind(alice)`. This branch writes `Parse(alice) Describe Sync`, and after its reply `Bind(alice)`, `Bind(bob)`. The stop in `advance()` is the half that fixes this case. **Why the flag was wrong.** Every `ReadyForQuery` sets `IS_READY_FOR_QUERY`, whichever Sync it answers. With two pipelined queries in flight, the first `ReadyForQuery` sets it while the second Sync is not answered. `pipelined_requests > 0` is the guard for that state. The `ErrorResponse` arm called `finish_request`, so the count went to 0 with a `ReadyForQuery` owed. A query that fails alone twice in a row reaches the same state: the second one is written behind the first in the gap of the first. **Tests.** `postgres-prepared-pipeline-reorder.test.ts` has 28 tests, 24 of them new. - Real server through the proxy, 17 tests that fail on main: 8 shapes of queries issued in the gap (a new statement and a prepared one, with equal and with different column types, through `execute()`, with a simple query, two new statements, three queries), each after a query that failed behind another one and after a query that failed alone twice in a row. One more on a connection reserved from a pool of 10. The shape with two new statements does not settle on main. - Real server, 5 tests that pass on main too: a transaction that begins in the gap, and 4 other kinds of failing query (a simple query, a Parse that fails with and without a parameter, a first execution without parameters). They pin that each kind leaves the queue at its `ReadyForQuery` and that the connection continues. - Mock, 2 tests that fail on main. PostgreSQL sends nothing but the `ReadyForQuery` after an `ErrorResponse`, and one `ReadyForQuery` for each Sync. With such a server, each half of the fix alone gives every query its own rows. So each half has a test with a peer that sends more. - The hold: the mock sends a row after the `ErrorResponse`. With only the hold reverted, 1 of 28 fails: the row goes to the query issued in the gap. No real peer that does this is known. - The stop: the mock replays the frames of pgdog. With only the stop reverted, 1 of 28 fails, and a debug build stops at the assertion `advance() passed a request that is not written yet`. **Measurements.** Base is main `a4f1429148`. PostgreSQL 17.11 on loopback, linux x64. - Edited arms on the success path: 0 executions in 1,703 fulfilled queries over 7 workloads (gdb counting breakpoints, debug build. Base 0 in 1,703). The calls of `advance()` from the `ReadyForQuery` arm and the Binds written at enqueue are equal on base and branch for the 6 workloads with a fixed order. The workload with a pool of 10 differs from run to run on both. - Steady state, 200 queries in bursts of 10: socket writes 20 (base 20), `advance()` entries 200 (base 200), reads 23 to 25 (base 23). - Instructions for each pipelined query: not measured. `perf` and `valgrind` are not in the container and have no package there. In place of that, gdb `stepi` on the release build with symbols, 20 calls in the middle of a burst: `advance()` 108 instructions for each call (base 108), `finish_request` 17 (base 17). - Release binary: `.text` +256 B. `advance()` +357 B, `on_result` +29 B, the two instances of `on()` -22 B each (`size`, `nm -S`). The debug assertion and the comments came after this measurement. They change no byte: `.text` and these symbol sizes are equal at `28aeaddf7` and `21c547a52`. - `size_of` `PostgresSQLQuery` 64 B (was 64), `PostgresSQLConnection` 672 B (was 672), `Flags` 7 B (was 7). - For each failing query: `finish_request` at the `ErrorResponse` 0 (was 1), at the `ReadyForQuery` 1 (was 0). The shape of the report: `advance()` entries 15 (was 15), socket writes 11 (was 12). - Counters when the connection closes, after 7 workloads with failing queries (50 connections): `pipelined_requests` 0, `nonpipelinable_requests` 0, `pending_requests` 0. - 12 shapes of queries in the gap, with the wire recorded: 0 wrong (was 6). Shapes that write before the held `ReadyForQuery`: 2 (was 12). The 2 are a prepared execute, which is still pipelined behind the failed request. After a failed simple query a prepared execute is written after the `ReadyForQuery` (was before). - The script of the report, release build: 0 of 100 rounds wrong in 3 runs (base 35, 37, 39. Official 1.4.2: 86, 84, 62). With the proxy: 0 of 50 (base 50, 1.4.2 50). - `close()` between the `ErrorResponse` and the held `ReadyForQuery`, 5 kinds of failing query and 6 ways to close: all 30 settle the same on 1.4.2 and here. A FATAL `ErrorResponse`, which the server follows with a close and no `ReadyForQuery`, also settles the same. **Existing suites**, debug build, local PostgreSQL: every `test/js/sql/postgres-*.test.ts`, `sql-prepare-false`, `sql-pool-transaction-isolation`, `sql-reserve-abort`, `sql-onconnect-onclose-throw`, `sql-close-pending-connection`, `wire-frames`. `postgres-string-leak` exceeds the 5 s timeout under the debug build, on base too. In the last run the container was under load, and subprocess tests in 4 other files passed 5 s. Each of those that I ran again alone passed. `sql.test.ts` against the local server: 18 to 20 of 854 fail, base 23. Each case that fails on the branch fails on base too. They need roles and an encoding that the local server does not have, or they depend on time under the debug build. The case for oven-sh#22395 passes. No run stopped at the new assertion. **A request that fails in its Parse** stays `Pending` with its place in `pending_requests`. The `discard_response` branch of the `ReadyForQuery` arm calls `finish_request`, whose `Pending` arm releases that place. `finish_request` has no edit. **Covered by reading only.** The failure path of `on_result` needs a command tag that fails to convert. It now keeps the request in flight when the `ReadyForQuery` is still to come. Before, the counter of that request was never released. **Not changed, named for later.** - A peer that sends a `ReadyForQuery` that answers no Sync releases a held request one message early. The rows stay in order (the stop). - The setter of `IS_READY_FOR_QUERY` and the early clear of `WAITING_TO_PREPARE` at `ParseComplete` and `ParameterDescription`. The held counters are the guard. A count of Syncs is the follow-up if the flag must be exact. It needs a rule for a `ReadyForQuery` that answers no Sync first: pgdog sends one, and 6 cases of `postgres-infinity-date.test.ts` answer one Sync with two. - Parse+Describe+Sync of a named statement and the first execution without parameters hold no counter. The flag pair guards them. - The arm of `advance()` for a request with no statement. It cannot be reached: `do_run` sets the statement before each push. - The MySQL adapter releases a request at a row it cannot decode, in the middle of its response (`JSMySQLConnection.rs`). oven-sh#43323 is open for that. **Open pull requests on the same code.** - oven-sh#35120 (re-prepare of invalidated statements) clears `IS_READY_FOR_QUERY` in the `ErrorResponse` arm for every error. That line alone closes the gap for a server that sends one `ReadyForQuery` for each Sync. It leaves the early release and the step-over. By reading, the pgdog case stays with that line alone. The two changes merge with no conflict. The retry branch of oven-sh#35120 releases and queues the request again at the `ErrorResponse`, so it needs the same hold after a rebase. - oven-sh#33370 (`Query.cancel()`) releases a `Pending` request whose Parse is on the wire and marks it `Fail`. That is the same early release from a second place. - oven-sh#43918 rewrites `encode_request`. This change does not touch it. **Self-review.** The concerns taken up: 1. The first text said that no open pull request covers this gap. oven-sh#35120 covers a part of it. The text above now says so. 2. oven-sh#33370 was not named. It is named above. 3. The sites of the same class that stay were not listed. They are listed above. 4. Each half was pinned only by a mock of a peer that does not exist. Addressed in part: the stop test now replays the frames of pgdog, and the run against the real pgdog is above. The hold test is still a mock. 5. `src/` had comments of more than one line. Each is one line now. 6. The reserved case ran on a pool of 1. It now reserves from a pool of 10: 10 sockets open and one carries the queries. 7. Nothing stated the write order in a debug build. `advance()` now asserts that it passes no request that is not written yet. **Credit.** The hold is the mechanism of oven-sh#43187. The stop is the arm of oven-sh#20986. </details> <!-- robobun:evidence:begin --> --- **[human-review]** gate passed · iteration 0 · 4 files touched <details><summary>fails on main (without fix)</summary> ```console ASAN without fix: 19 FAILED $ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/js/sql/postgres-prepared-pipeline-reorder.test.ts bun test v1.4.3 (367d939) test/js/sql/postgres-prepared-pipeline-reorder.test.ts: Container ready via docker-compose: postgres_plain at 127.0.0.1:5432 (pass) postgres > a prepared-statement execute does not jump a queued unwritten prepare [950.35ms] (pass) postgres > prepared executes do not jump multiple queued unwritten prepares [371.61ms] (pass) postgres > mixed burst of prepared and new statements returns every row in order [369.38ms] (pass) postgres > a prepared-statement execute does not jump a queued unwritten simple query [128.05ms] Container ready via docker-compose: postgres_plain at 127.0.0.1:5432 204 | return `{${values.map(arrayValueSerializer.bind(this, type, isPostgresNumericType(type), isPostgresJsonType(type))).join(delimiter)}}`; 205 | } 206 | function wrapPostgresError(error) { 207 | if (Error.isError(error)) { 208 | return error; 209 | return new PostgresError(error.message, error); ^ PostgresError: Failed to read data hint: "The quer ... (truncated) release without fix: all passed bun test v1.4.3-canary.1 (21c547a) test/js/sql/postgres-prepared-pipeline-reorder.test.ts: Container ready via docker-compose: postgres_plain at 127.0.0.1:5432 (pass) postgres > a prepared-statement execute does not jump a queued unwritten prepare [54.98ms] (pass) postgres > prepared executes do not jump multiple queued unwritten prepares [48.48ms] (pass) postgres > mixed burst of prepared and new statements returns every row in order [24.94ms] (pass) postgres > a prepared-statement execute does not jump a queued unwritten simple query [12.04ms] Container ready via docker-compose: postgres_plain at 127.0.0.1:5432 (pass) postgres > before the ReadyForQuery of a query that failed behind another query > a new statement with other column types, a prepared one resolve with their own rows [131.62ms] (pass) postgres > before the ReadyForQuery of a query that failed behind another query > a new statement, a prepared one, through execute() resolve with their own rows [141.14ms] (pass) postgres > before the ReadyForQuery of a query that failed behind another query > a new statement, a simple query resolve with their own rows [142.52ms] (pass) postgres > before the ReadyFor ... (truncated) ``` </details> <details><summary>passes on PR (with fix)</summary> ```console ASAN with fix: all passed $ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/js/sql/postgres-prepared-pipeline-reorder.test.ts bun test v1.4.3 (367d939) test/js/sql/postgres-prepared-pipeline-reorder.test.ts: Container ready via docker-compose: postgres_plain at 127.0.0.1:5432 (pass) postgres > a prepared-statement execute does not jump a queued unwritten prepare [609.58ms] (pass) postgres > prepared executes do not jump multiple queued unwritten prepares [492.29ms] (pass) postgres > mixed burst of prepared and new statements returns every row in order [253.06ms] (pass) postgres > a prepared-statement execute does not jump a queued unwritten simple query [157.56ms] Container ready via docker-compose: postgres_plain at 127.0.0.1:5432 (pass) postgres > before the ReadyForQuery of a query that failed behind another query > a new statement, a prepared one resolve with their own rows [1856.81ms] (pass) postgres > before the ReadyForQuery of a query that failed behind another query > a new statement with other column types, a prepared one resolve with their own rows [1664.26ms] (pass) postgres > before the Rea ... (truncated) release with fix: all passed $ bun scripts/build.ts --profile=release [configured] bun-profile → bun (stripped) in 7581ms (unchanged) ninja: Entering directory `/workspace/bun/build/release' [1/186] gen cpp.rs (cppbind) [2/186] gen generated_host_exports.rs generated_host_exports.rs: 121 exports (host=5, lazy=10, generic=106, rust=0); 248 extern-C blocks audited [3/186] gen JS modules (bundle-modules) Preprocess modules (11769ms) Bundle modules (1818ms) Postprocesss modules (245ms) Bundle Functions (1012ms) Generate Code (63ms) [14.93s] Bundled "src/js" for production 2613 kb 198 internal modules 13 native modules 50 internal functions across 16 files [4/74] cxx obj/unified/UnifiedSource-src_jsc_bindings-1.cpp.o [5/74] cxx obj/unified/UnifiedSource-src_jsc_bindings-3.cpp.o [6/74] cxx obj/unified/UnifiedSource-src_jsc_bindings-0.cpp.o [7/74] cxx obj/unified/UnifiedSource-src_jsc_bindings-5.cpp.o [8/74] cxx obj/unified/UnifiedSource-src_jsc_bindings-4.cpp.o [9/74] cxx obj/src/jsc/bindings/bindings.cpp.o [10/74] cc obj/packages/bun-usockets/src/loop.c.o [11/74] cxx obj/unified/UnifiedSource-src_jsc_bindings_node-0.cpp.o [12/74] cc obj/packages/bun-usockets/src/bsd.c.o [13/74] cxx obj/un ... (truncated) ``` </details> <details><summary>diff hotspot</summary> ``` src/sql_jsc/postgres/PostgresSQLConnection.rs | 17 +- src/sql_jsc/postgres/PostgresSQLQuery.rs | 16 +- .../sql/postgres-prepared-pipeline-reorder.test.ts | 278 ++++++++++++++++++++- test/js/sql/wire-frames.ts | 56 +++++ 4 files changed, 355 insertions(+), 12 deletions(-) ``` </details> **gate history** · 1 passed · 0 rejected · iteration 0 <details><summary>evidence per changed file</summary> ``` file reads edits tests src/sql_jsc/postgres/PostgresSQLConnection.rs 5 0 39 src/sql_jsc/postgres/PostgresSQLQuery.rs 3 0 39 test/js/sql/postgres-prepared-pipeline-reorder.test.ts 2 0 38 test/js/sql/wire-frames.ts 2 2 40 ``` </details> <!-- robobun:evidence:end -->
…ven-sh#43323) ### Problem - `Bun.SQL` (MySQL): when the client cannot decode a row, its query rejects and the rest of the response goes to the next query. After a `CALL` with two result sets, the next query resolves with `[{"leftover":777}]`, or the connection fails with `ERR_MYSQL_UNEXPECTED_PACKET`. - `JSMySQLConnection::on_result_row` (`src/sql_jsc/mysql/JSMySQLConnection.rs`) marked the query `Fail` at the bad row. `advance()` pops a `Fail` head at the next terminator, while the server still answers it. - A JSON column that `JSON.parse` refuses fails in `to_js`, which returned `JSError`. That closes the connection: every queued query rejects with that `SyntaxError`. A default MariaDB stores such text (`{"name":"O\'Brien"}` passes `JSON_VALID`). ### Fix - The query rejects at once but stays at the queue head, because the server still sends the rest of its response. The new `DISCARD_RESPONSE` flag skips those rows. The last terminator, a server error, or a close marks it `Fail`, with no second rejection. - A `to_js` error takes the same path. A pending termination exception still fails the connection. - Verified: `test/js/sql/sql-mysql-row-decode-error.test.ts` (9 mock tests, 1 MariaDB test, all fail without the fix). Also `test/js/sql/sql-mysql*.test.ts`. - Self-reviewed: 9 concerns raised, 7 addressed. Not taken: a merge into oven-sh#43187, the same fix for Postgres (linked instead), and issues for the "Not covered" sites. ### Background - `Bun.SQL` pipelines: it writes several queries before the first reply. Each reply packet goes to the head of a FIFO queue. - A response is one or more result sets. Each ends with a terminator. Its `SERVER_MORE_RESULTS_EXISTS` flag says that another follows. - `advance()` pops finished requests from the queue head and writes queued ones. <details><summary>Notes</summary> **Reproduction.** Mock server, 1.4.3 (`b52d51348`), `max: 1`, before the fix: ``` A (decode error, two result sets) bad query : REJECTED ERR_MYSQL_INVALID_BINARY_VALUE: Failed to bind query neighbour : [{"leftover":777}] B (to_js error) bad query : REJECTED SyntaxError: JSON Parse error: Expected '}' neighbour : REJECTED SyntaxError: JSON Parse error: Expected '}' ``` **MariaDB.** A JSON column is `LONGTEXT` with a `JSON_VALID` check constraint. On MariaDB 11.8.6 with default settings, `JSON_VALID` accepts `\'`, `\x41`, `\q` and a Windows path with single backslashes (`{"path":"C:\Users\qux"}`). `JSON.parse` refuses all of them. Before the fix, one such row closed the connection: a pipelined query that the server had answered rejected with `SyntaxError: JSON Parse error: Invalid escape character '`, and a `TEMPORARY` table was gone for the next query. Before v1.4.0 such a column came back as a string (oven-sh#37130 began to parse it). The MariaDB test in the new file covers this. It runs where `mariadb_plain` is provided (the same gate as `sql-mariadb-json.test.ts`). MySQL has a native JSON type and sends normalized text, so only a mock reaches this path there. The decode error (`row.decode`) needs a mock: a healthy server does not send a binary `DATE` of 5 bytes. MySQL frames each row in a packet, and `process_packets` skips the packet whole, so the stream stays in step after any row decode error. **Not covered, on purpose.** - A column definition that cannot be decoded, at prepare time (`handle_prepared_statement`) or at execute time (`handle_result_set`), still fails the connection. A column type that `FieldType` does not know reaches these sites with `ERR_MYSQL_UNSUPPORTED_COLUMN_TYPE`, for example `MYSQL_TYPE_VECTOR` (0xf2) from MySQL 9. The fix there is to decode an unknown type as bytes, not to reject the query, so it is a separate change. - A row whose lengths run past its packet still reads as `ShortRead`, which means "wait for more data". oven-sh#35118 bounds reads to the packet. After it, such a row takes the path of this PR. - A result set header, `OK`, `EOF` or `ERR` packet that cannot be decoded still fails the connection. The stream is out of step then. **Overlap with open PRs.** oven-sh#33190 adds `mysqlBinaryResultSetRow` and `mysqlBinaryResultSet` to `test/js/sql/wire-frames.ts` under the same names. The versions here take one more optional argument (`statusFlags`). The PR that lands second drops its copy. `settle()` in the new test file is the helper from the test file of oven-sh#43187. The PR that lands second can share it. **Accounting.** `mark_current_request_as_finished` is no longer called at the bad row. The last terminator, or the `ERR` packet, calls it while the request still counts as running, so `pipelined_requests` or `nonpipelinable_requests` drops once. A build that skips this release for a discarding request makes 8 of the 9 mock tests time out: a statement that is not prepared yet, and a `.simple()` query, can only start after the counter drops. **Other shapes I ran by hand on the ASAN debug build.** - Real MariaDB: a 200000 row result whose first row is not JSON, with `Bun.gc(true)` in the rejection handler while the rest streams. The same with the bad row second to last. - Real MariaDB: `sql.begin()`. The callback throws on the rejection and `ROLLBACK` runs on the same connection. A callback that catches the rejection continues in the same transaction. - Real MariaDB: `.values()` rejects the same way. `.raw()` does not decode and resolves. - Mock: the legacy framing without `CLIENT_DEPRECATE_EOF`. - Mock: the server drops the connection, or `sql.close({ timeout })` fires, while the response is discarded. The query rejects once and the next one rejects with `ERR_MYSQL_CONNECTION_CLOSED`. **Test runs** on the debug build against a local MariaDB: `sql-mysql.test.ts` (plain image only) gives 97 pass and 7 fail, `sql-mysql.transactions.test.ts` gives 12 pass and 2 fail. The same tests fail on the build without the fix. They compare MySQL error texts that MariaDB words differently, or they exceed their timeout under ASAN. `BUN_JSC_validateExceptionChecks=1` passes on the new file. </details>
Wait for the Node HTTP listener and all 32 body-validated fetches before disposing the server. Replace the unjoined timer and vacuous Infinity comparison with isolated Response and AbortSignal reclamation checks. The original fixture fails with ERR_SERVER_NOT_RUNNING; a retained-response control fails the finite object bound.
Start renegotiation only after the first client handshake callback sends a framed ready message. Keep the framed payload acknowledgement, exact two-handshake assertion and pauseOnConnect regression check. This removes a setup race without changing TLS runtime code.
Import the PostgreSQL and MySQL query-isolation fixes as an additional merge. Preserve the published OpenClaw WebKit pin and all previously qualified fork changes; append the new upstream target to the changelog.
steipete
changed the title
steipete
marked this pull request as ready for review
October 5, 2026 08:35
This was referenced Oct 5, 2026
steipete
added a commit
that referenced
this pull request
Oct 5, 2026
OpenClaw's catalog performance consumers reach `HeapProfiler.startSampling`, which the in-process inspector did not implement. Add `startSampling`, `getSamplingProfile`, and `stopSampling` for callback and promise Sessions using the real per-VM JSC allocation sampler from the immutable [OpenClaw WebKit release](https://github.com/openclaw/WebKit/releases/tag/autobuild-42ab38d705d4838748ccee77e7deb0e4e35515ee) integrated by #124. Preserve Node defaults, synchronous callbacks, repeated-start behavior, major/minor collection flags, worker isolation, and session cleanup. Profile retrieval forces full collection. Numeric validation uses an immutable intrinsic and truncation happens in the native binding, so replacing public numeric helpers cannot violate the engine's interval precondition. Sampling through a worker's `connectToMainThread()` remains unsupported. Upstream's CLI heap-profile work in oven-sh#34842 is separate from these Session commands. Validation on Linux x64, with unmodified OpenClaw `b02eab15853b6471d1869f2a2cb0f135b7aefe4d` and Node 24.21.0 controls: - Inspector suite: 78 passed, zero failures. - Plain fork selection: 25 files, 27/27 result rows passed. - Gateway and Codex native catalog performance consumers: each passes on the candidate and Node; the earlier baseline fails both at missing `startSampling`. - The original sampler patch fails both public numeric-helper override cases; the corrected candidate and Node pass. - Pinned formatting checks and independent scoped P2 review pass. The committed production manifest is unchanged by this PR and selects the published engine. No runtime publication is included. Final head `c8de1bfd252a50c03286cb83d7e3a73cca6b5395` passes [both native CI lanes](https://github.com/openclaw/bun/actions/runs/37292758243), formatting, source lints, and built-in module typechecking. The complete standalone Linux proof used `46754a481e350a63b64ac3edd5449ef1056604ba`; the final follow-up adds only the missing TypeScript declaration for the intrinsic, with unchanged runtime code. The final head also builds successfully on macOS arm64.
steipete
added a commit
that referenced
this pull request
Oct 5, 2026
Connect Bun's stack reporting and transpiler to the syntax positions in the published OpenClaw WebKit engine integrated by #124. Calls, property reads, constructors, and async continuations use the engine's selected locations; returning built-in error constructors retain their frames and `new` syntax. Runtime transpilation preserves callee parentheses and computed access and maps call, bracket, and template delimiters back to the original source. Bump the runtime transpiler cache format to 38 so cached output from earlier versions cannot retain the old positions. Keep existing file-name and eval-origin formatting. Update the ErrorEvent inspection caret to the `new` token confirmed by Node. This continues the source-position work in oven-sh#35179, oven-sh#37396, and oven-sh#41580; thanks @robobun. Native macOS arm64 validation on candidate `8b774531c290dac5eeb35cd62b7bce517676a2e5`: - Stack, util, and source-map suites: 343 passed. - Minifier suite: 68 passed. Inspection suite: 95 passed, one existing skip. - Runtime cache replacement and marked cached-output replay pass at version 38, preserving all 56 coordinate assertions. - Rust checks: all 12 targets pass, with zero skips. Independent scoped P2 review and source lint pass. Unmodified OpenClaw `b02eab15853b6471d1869f2a2cb0f135b7aefe4d` consumers pass on the candidate and Node 24.21.0: SDK alias 67 passed with one existing skip; ordered loader pair 8 passed, with execution order, shared worker and canonical test policy verified. The plain Darwin selection has zero new regressions on the local macOS 27 host: candidate 20/21 rows versus baseline 19/21. The candidate's only failed file contains two socket-cancellation assertions reproduced unchanged with the pre-stack binary in isolation and the full selection. A baseline-only HTTP failure did not recur on the candidate. The fork-pinned Node 26.3.0 harness is used; the OpenClaw consumer controls use Node 24.21.0. No assertions or selections were weakened. Both exact-head native CI lanes pass in [run 37298205240](https://github.com/openclaw/bun/actions/runs/37298205240). The cache oracle improves from 2/56 matching positions on the baseline to 56/56 on both candidate runs. The committed production WebKit manifest remains unchanged. Formatting, source lints, JavaScript typechecking, Clippy, Miri and cargo tests pass. The Rust workflow succeeds; its explicitly advisory mordant job reports the unchanged preexisting bare-bool-argument style finding in `src/resolver/package_json.rs`, with no suppression or baseline change in this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This merges oven-sh/bun through
9bd19c98eacc01530a4e7609bc427abffa87d77eand adopts immutable OpenClaw WebKit42ab38d705in the same change. The newer engine provides upstream's idle compiler-thread andAtomics.waitmemory release. Its namespace API also requires the Bun counterpart from #106, which is retained as a distinct credited prerequisite commit.The import/require namespace change removes the former prototype/private-field workaround: imports remain unmarked, while
require(esm)receives its own marker and live exports. The existing #106 implementation and regressions are preserved. Thanks @robobun for the upstream namespace work adapted in oven-sh/bun#33894 and #39888.Extended namespace qualification exposed the existing async CJS package-context bug already addressed by #117. This branch preserves that original commit as a merge prerequisite: the file-format parser hint and package interop tag remain separate. The old baseline and first candidate both reproduce the incorrect bare default; Node preserves the exports object. The original assertions remain intact.
The original two textual conflicts preserve the fork's checksum manifest resolver and event-loop idle accounting. The scalar-only
bun:package-import exception, resolver/cache fixes, module-loader context, CallSite work, macOS cache serialization and current VM cached-data fixes remain intact. The changelog retains current main as an exact prefix. The complete published manifest is committed unchanged; all nine artifact digests, sizes and target names were independently checked against the immutable release. There is no upstream-artifact fallback.The TLS nightly update is retained as an additional merge commit, preserving the previous candidate and upstream history. Its TLS setup now runs once per connection, avoiding repeated setup when a pooled socket is reused during renegotiation. The only additional conflict joins the TLS test helpers, preserving the fork default-CA coverage. The engine target is unchanged.
Qualification is complete on head
65a7af856178f994b5319404c3602ae71d991980. The final upstream delta contains PostgreSQL/MySQL query-isolation fixes, imported unchanged in an additional merge. WebKit, HTTP and module-loader paths are unchanged by that delta. Its direct SQL proof passes all 38 tests, including real PostgreSQL and MariaDB containers:The final CI merge tree matches the reviewed candidate exactly. Format, JavaScript/source lints, Clippy and Miri pass. The explicitly advisory Mordant check reports the existing
resolve_targetboolean-argument warning in an unchanged resolver blob.The initial consumer matrix had zero regressions, but both Bun binaries hit the same packaged Gateway setup failure before its execution assertions. Restoring the isolated runner’s omitted Linux user-bus environment makes all 20 files pass on Node, fork main and the prior candidate with default publication checks intact. The latest canary consumer replay also passes all 20 files: 474 passed and one expected Windows-only skip, with zero regressions. Both grouped and individual Vite production builds pass on the paired-engine candidate and Node; fork main reproduces the missing
zod.zexport in both modes. The unchanged memory, namespace and resolver-oracle proof remains applicable to the later TLS and SQL deltas.The expanded HTTP selection exposed one inherited fixture failure on both platforms:
fetch-leak.test.tsreturned before its server listened or its sampling timer completed, then disposal raisedERR_SERVER_NOT_RUNNING. The test-only repair awaits readiness and 32 body-validated requests, joins the child, and checks finite Response/AbortSignal object counts. The CI-built Darwin executable passes the repair; the original fixture reproduces the failure, and a deliberate retained-response control fails with 33 live objects. The repaired full leak file passes all 31 tests on Linux baseline and candidate and passes Linux CI. A subsequent Darwin run exposed a separate inherited TLS fixture race: its server could start renegotiation before the client observed its initial handshake. A framed client-ready message now establishes that ordering, while retaining the exact two-handshake assertion and acknowledgement-gated payload. The full 21-test file and ten focused Darwin repetitions pass; deliberately pausing again on the second handshake still fails. Both lanes passed on the preceding head, with one unchanged-head Darwin retry after a VM timeout timing observation. The full VM file passed five times on each of the verified fork-main and candidate Darwin binaries; the assertion and timeout implementations were unchanged. That failure remains recorded, not claimed fixed. Both full CI lanes pass on the final SQL-sync head.Landing uses a merge commit, preserving upstream history. No Bun release or tag is part of this PR. Remaining inspector, accounting and stack adapter PRs stay separate. Supersedes #106 only when this PR merges.