bun install: a streamed tarball fails when a body chunk ends inside a tar sparse map or a large pax header #44675

What happens

bun install extracts a tarball while it downloads when the response has no Content-Length, or when the tarball is 2 MiB or larger. In that mode the install fails if a chunk of the response body ends inside some parts of the header of a tar member. The tarball is valid. The buffered extractor installs it (a file: tarball, or BUN_FEATURE_FLAG_DISABLE_STREAMING_INSTALL=1).

The first two rows are the output of the script below. The last two rows are from hand-built tarballs in the tests on the branch named under Status.

From the code only, not run: on macOS the data of an AppleDouble ._name member is read in the same way (read_mac_metadata_blob, message Truncated archive detected while reading macOS metadata).

I know of no user report of this failure. I found it while I wrote sparse-member tests for #36542. Those tests send each sparse member in one piece for this reason.

Reproduce

// bun repro.mjs [gnu|posix]        needs GNU tar
import { $ } from "bun";
import { createHash } from "node:crypto";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { gzipSync } from "node:zlib";

const format = process.argv[2] ?? "gnu";
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "sparse-repro-"));
for (const d of ["src/package", "tmp", "cache", "app"]) fs.mkdirSync(path.join(dir, d), { recursive: true });
fs.writeFileSync(path.join(dir, "src/package/package.json"), JSON.stringify({ name: "sparse-pkg", version: "1.0.0" }));
// A sparse file: seven 512-byte extents in 300000 bytes. tar only maps a file that has holes on disk.
const fd = fs.openSync(path.join(dir, "src/package/m.bin"), "w");
for (const at of [0, 8192, 16384, 24576, 32768, 40960, 299_488]) fs.writeSync(fd, Buffer.alloc(512, "x"), 0, 512, at);
fs.closeSync(fd);
await $`tar --sparse --hole-detection=raw --format=${format} -cf ../pkg.tar package/package.json package/m.bin`.cwd(path.join(dir, "src"));

const tar = fs.readFileSync(path.join(dir, "pkg.tar"));
const tgz = gzipSync(tar, { level: 0 }); // stored: byte n of the tar is byte 15 + n of the .tgz
// The header block of the sparse member. gnu: the sparse extension block follows it.
// posix: the member data follows it, and starts with the sparse map.
const header = Math.floor(tar.indexOf(format === "gnu" ? "package/m.bin" : "GNUSparseFile") / 512) * 512;
const cut = 15 + header + 512 + 3;
const integrity = "sha512-" + createHash("sha512").update(tgz).digest("base64");
const tmp = path.join(dir, "tmp");

let exited = false;
const server = Bun.serve({
  port: 0,
  async fetch(req) {
    if (!req.url.endsWith(".tgz")) {
      const dist = { integrity, tarball: `${server.url}sparse-pkg/-/sparse-pkg-1.0.0.tgz` };
      return Response.json({ name: "sparse-pkg", "dist-tags": { latest: "1.0.0" }, versions: { "1.0.0": { name: "sparse-pkg", version: "1.0.0", dist } } });
    }
    return new Response(
      new ReadableStream({
        type: "direct",
        async pull(c) {
          c.write(tgz.subarray(0, cut));
          await c.flush();
          // Send the rest after the extractor has used up the first piece.
          while (!exited && !fs.readdirSync(tmp).some(n => n.endsWith(".sparse-pkg"))) await Bun.sleep(5);
          await Bun.sleep(100);
          c.write(tgz.subarray(cut));
          await c.flush();
          c.close();
        },
      }),
    );
  },
});
fs.writeFileSync(path.join(dir, "app/package.json"), JSON.stringify({ name: "app", dependencies: { "sparse-pkg": "1.0.0" } }));
fs.writeFileSync(path.join(dir, "app/bunfig.toml"), `[install]\nregistry = "${server.url}"\n`);
const proc = Bun.spawn({
  cmd: [process.execPath, "install"],
  cwd: path.join(dir, "app"),
  env: { ...process.env, BUN_TMPDIR: tmp, TMPDIR: tmp, BUN_INSTALL_CACHE_DIR: path.join(dir, "cache"), BUN_INSTALL_STREAMING_DRAIN_THRESHOLD: "1" },
  stdout: "inherit",
  stderr: "inherit",
});
proc.exited.then(() => (exited = true));
console.log("exit code:", await proc.exited);
server.stop(true);
fs.rmSync(dir, { recursive: true, force: true });

bun repro.mjs gnu and bun repro.mjs posix print the messages above and exit code 1.

The script serves one tarball in two pieces. The first piece ends 3 bytes after the header block of the sparse member. BUN_INSTALL_STREAMING_DRAIN_THRESHOLD=1 makes the extractor start on the first piece. With the default of 256 KiB a chunk can only end there in a larger tarball.

Cause

The streaming extractor (src/install/TarballStream.rs) feeds libarchive from a read callback that answers ARCHIVE_RETRY when no bytes are buffered. patches/libarchive/nonblocking-read.patch lets the tar reader resume a header at two points: the read of a 512-byte header block, and an extension header whose payload is at most 1 MiB. Every other read of the header phase takes "no bytes yet" for a truncated archive:

• readline and gnu_sparse_10_read (PAX 1.0 sparse map and its padding)
• gnu_sparse_old_read (old GNU sparse extension blocks)
• read_mac_metadata_blob (AppleDouble data, macOS builds)
• header_pax_extension and the pax_attribute readers, for a payload over 1 MiB

Status

A fix is on the branch robobun/399843a7/tar-sparse-map-resume. It is not a pull request. With a new nonblocking tar option the reader keeps the bytes of a header in its buffer until it has parsed the whole header, and parses the header again after a read that ran out of input. On my set of 184 archives, a harness that splits the input at every byte offset found no difference from a single read.

A review of the branch asked for a maintainer decision before a pull request, because no report of this failure exists. It also found these open points, some of them on archives outside that set:

• The fix must land after tar extraction: no disk allocation from the entry header, file length from libarchive #36542 or Keep the file size when preallocating a write #44425. On main the streaming extractor writes a sparse member that ends in a hole with the wrong length (41472 bytes for a 300000-byte file in my test). Today a chunk that ends inside the map of such a member fails the install. With only this fix the install succeeds with the short file.
• After a read that ran out of input, the branch restores four fields of the tar reader. It must restore all scalar header state, archive_format included.
• The branch removed the read-ahead of an extension payload. It must keep one, or a pax record with a key over 512 bytes is skipped with a warning when the payload comes in pieces.
• Each new piece parses the header again from its start. For a sparse map of 30 MiB that is 120 parses at the default drain threshold. The wait for new input must grow geometrically.
• The gzip filter has one more place with the same failure: a gzip header whose last optional field is FEXTRA.