Public Read/Write I/O API + pub TurboQuantIndex::from_parts

[gburd]

Context

I'm the author of pg_turbovec, a PostgreSQL extension that uses turbovec as the kernel for indexed cosine / inner-product / L2 / L1 ANN search at storage sizes 10–20× smaller than pgvector. It's been carrying a small additive vendor patch on top of turbovec 0.5.0/0.6.0 to handle two needs that database extensions hit but the upstream API doesn't currently support:

1. In-memory deserialisation. When the index payload is already in RAM (read out of a bytea column or a buffer-manager-pinned page), going through IdMapIndex::load(path) requires a tmpfile round-trip — write to /tmp, fsync, re-open, read back, parse. On a 1 M × 384-d × 4-bit index that's ≈195 MB of needless I/O. The internal parsers in src/io.rs are already generic over Read/Write; widening the public surface to expose that internal generality is a 100-line additive change.

2. Constructing an IdMapIndex from already-decoded parts. Database storage backends sometimes have their own page format (e.g. PostgreSQL relfiles) and want to skip the .tvim wire format entirely. Promoting TurboQuantIndex::from_parts (and the small packed_codes() / scales() accessors it pairs with) from pub(crate) to pub lets external embedders construct the index from their own deserialised bytes.

Proposed API additions

All additive; no behavioural change to existing APIs; no wire-format change.

// src/io.rs
pub fn write_to<W: Write>(w: &mut W, …) -> io::Result<()>
pub fn load_from<R: Read>(r: &mut R) -> io::Result<…>
pub fn write_id_map_to<W: Write>(w: &mut W, …) -> io::Result<()>
pub fn load_id_map_from<R: Read>(r: &mut R) -> io::Result<…>

// src/id_map.rs
impl IdMapIndex {
    pub fn write_to_writer<W: Write>(&self, w: &mut W) -> io::Result<()>
    pub fn load_from_reader<R: Read>(r: &mut R) -> io::Result<Self>
}

// src/lib.rs — three previously-pub(crate) items become pub
pub fn TurboQuantIndex::from_parts(dim, bit_width, n_vectors, packed_codes, scales) -> Self
pub fn TurboQuantIndex::packed_codes(&self) -> &[u8]
pub fn TurboQuantIndex::scales(&self) -> &[f32]

Why this matters in the field

pg_turbovec measured a 21× cold-scan speedup (26 310 ms → 1 256 ms p50, 1 M × 1536-d real OpenAI ada-002 corpus on a single bench host) by combining:

• The Read/Write API to read TVIM payloads straight from PG's buffer manager (no tmpfile).
• pub fn from_parts to construct the IdMapIndex from the buffer-pinned bytes, skipping deserialisation entirely.
• Persisting the SIMD-blocked layout into the relfile at ambuild time so the per-backend pack::repack (12–15 s on this corpus) becomes amortised constant.

The first two bullets are the API change in this issue; the third lives entirely in pg_turbovec. Without the upstream changes, pg_turbovec carries the vendor patch indefinitely, which is fine but adds a maintenance step every time turbovec releases.

Status of a draft PR

I respect the by-invitation policy in CONTRIBUTING. I've drafted the implementation against v0.6.0 on a branch in my fork — gburd/turbovec in-memory-io-and-pub-from-parts — so if you're amenable to the change and would like to invite a PR, the diff is already ready (3 files, +107 / -11 lines, all upstream cargo test --tests --release passes). Happy to take it on if you prefer; equally happy if you'd rather pick this up yourself or not at all.

Test results on the draft branch

test result: ok. 6 passed   tests/codebook
test result: ok. 5 passed   tests/concurrent_search
test result: ok. 5 passed   tests/distortion
test result: ok. 4 passed   tests/encode
test result: ok. 15 passed  tests/filtering
test result: ok. 11 passed  tests/id_map
test result: ok. 5 passed   tests/io_versioning
test result: ok. 6 passed   tests/kernel_correctness
test result: ok. 21 passed  tests/lazy_init

[gburd]

main...gburd:turbovec:in-memory-io-and-pub-from-parts

[faysou]

Hi @gburd I've worked on a cache as well as described in this issue #68 maybe some ideas could be useful for your use case. A lot of things can be cached. I'm not saying my PR should be merged, it just works for me for now.

[ZeyadMohamad]

Hello @gburd @faysou
I went ahead and implemented this proposal against current main (0.9.0). Since @gburd 's draft was written for v0.6.0, I rewrote it fresh so it carries the TQ+ v3 fields and keeps all of the #108 untrusted-load hardening underneath the new entry points. Please take a lot at my fork branch and see if my approach might help in any way.

[gburd]

---

Update from the pg_turbovec side, now that the fork tracks v0.9.0.

Thanks @ZeyadMohamad for re-implementing this against current main
with the TQ+ v3 fields and the #108 hardening — that's exactly the
right basis, and it's much more useful than my original v0.6.0 draft.
And thanks @faysou; the #68 caching work is complementary.

One scope note that should make a PR smaller and easier to accept: of
the three asks in the original issue, pg_turbovec now only needs
the first.

1. pub TurboQuantIndex::from_parts + packed_codes() / scales()
   accessors — still needed, and the single highest-value item. Any
   database-storage embedder (we use PostgreSQL relfile pages) wants to
   construct an index from already-decoded bytes without a tmpfile
   round-trip. pg_turbovec's entire cache-fill path is built on this.

2. from_id_map_parts* constructors on IdMapIndex — we carry these
   in our fork, but they're arguably pg_turbovec-specific (they thread
   pre-baked SIMD-blocked layout + rotation + codebook through, which is
   our relfile-resident optimisation). I'm not sure they belong upstream
   unless another embedder wants the same "skip prepare on load" path.
   Happy to discuss; they may be better left as a fork concern.

3. The Read/Write IO API (write_to/load_from etc.) — we
   no longer need this. pg_turbovec reads straight from PG buffer
   pages via from_parts, never from .tv/.tvim files, so the
   streaming-IO half of the original issue is moot for us. It's only
   worth doing if another consumer wants in-memory .tvim (de)serialisation.

So if you're inclined to take anything, #1 alone (the pub from_parts

• accessors that @ZeyadMohamad already implemented) closes the issue for
  pg_turbovec's purposes. The rest is optional.

Separately — thank you for #108. The pre-AVX2 scalar-fallback fix (the
perm0 de-interleave) was a silently-wrong-top-k bug that pg_turbovec
hit in production on a pre-AVX2 Xeon bench host; upgrading our fork to
v0.9.0 fixed it. The audit was real-world load-bearing for us.

[ZeyadMohamad]

Thanks @gburd , your clarification helped a lot.
The PRs are currently restricted/closed, would you be open to inviting me to open this PR against upstream? I’d really like to contribute this properly to the project instead of carrying it only in my fork.
I can either push my current work as-is to a branch for review or directly to the main branch, or open a smaller PR strictly limited to making TurboQuantIndex::from_parts public and exposing the related packed_codes() / scales() accessors, since that seems to be the only part pg_turbovec still needs.
@gburd @RyanCodrai @faysou

[RyanCodrai]

Hi Folks, sorry for not being as attentive as I would like to be on some of these issues. Currently travelling in SF for 2 weeks but will try to find time

[RyanCodrai]

Somebody’s AI agent having a fit?

[ZeyadMohamad]

I want to upstream everything really, but i have no privilege to do so. I am not one of the current contributors and no PR access was given to me. In the image below it shows that PRs are all restricted