Skip to content

Feedback from ZstdLib #63

Closed
Closed

Description

@tecosaur

Hey Jakob,

As you know, I've been working with Claude on ZstdLib, and BufferIO has worked pretty well overall (6x the line iteration speed vs. TranscodingStreams is a nice headline result, and it's trim-compatible!).

That said, a few slightly less smooth spots turned up, and so I've lazily asked for them to be written up into an issue, which I've just pasted below:


🤖 Fable 5.2

Faster readuntil, typed reads, and bulk reads on AbstractBufReader

While benchmarking I came across three places where the generic reader methods leave some speed on the table, and thought the measurements might be useful to you.

All numbers are from BufferIO 0.3.2 on Julia 1.13.2 (i5-13600K, one pinned core), taking the fastest of ten 0.2 s runs. The script at the end reproduces the first two sections using BufferIO alone, and checks that both prototypes return exactly what BufferIO returns today across buffer sizes from 1 to 4096 bytes.

1. readuntil builds a VecWriter on every call

readuntil(x, delim) creates a VecWriter and hands it to copyuntil. Constructing the writer takes about 12 ns (it allocates a 32-byte string-backed vector), against 6 ns for Vector{UInt8}(undef, 32), and the bytes then go through the general write path.

When the delimiter is already in the buffer, which it is for most short fields, the bytes can be copied out directly. That is the same shape as the fast path readline already has:

function Base.readuntil(x::AbstractBufReader, delim::UInt8; keep::Bool = false)
    buffer = get_nonempty_buffer(x)::Union{Nothing, ImmutableMemoryView{UInt8}}
    buffer === nothing && return UInt8[]
    pos = findfirst(==(delim), buffer)
    pos === nothing && return readuntil_slowpath(x, delim, keep)  # today's implementation
    v = Vector{UInt8}(@inbounds MemoryViews.truncate(buffer, pos - !keep))
    @inbounds consume(x, pos)
    return v
end

Splitting 8 MiB of JSON lines on commas (400,001 calls):

Reader ns per call
Base, IOBuffer 37.2
BufferIO today, CursorReader 20.9
Direct copy, CursorReader 12.8
BufferIO today, BufReader(IOBuffer) 21.0
Direct copy, BufReader(IOBuffer) 12.7

BufferIO is already well ahead of Base here, so this is an improvement on something good. Downstream it matters a little more than the table suggests: on a zstd stream it takes readuntil from 532 to 846 MB/s, which moves it from behind CodecZstd (739 MB/s) to ahead of it.

2. Typed reads on the abstract types

read(reader, UInt32) on an AbstractBufReader is a MethodError, and so is write(writer, 0x01020304) on an AbstractBufWriter. Both work through IOReader and IOWriter, though that route goes through Base's generic machinery. Since the reader already exposes its buffer, a value that lies wholly within it can be loaded directly:

function Base.read(x::AbstractBufReader, ::Type{T}) where {T <: Union{Int16, UInt16, Int32, UInt32, Int64, UInt64, Float32, Float64}}
    buffer = get_buffer(x)::ImmutableMemoryView{UInt8}
    length(buffer) < sizeof(T) && return read_slowpath(x, T)  # refill, or copy across a buffer boundary
    value = GC.@preserve buffer unsafe_load(Ptr{T}(pointer(buffer)))
    @inbounds consume(x, sizeof(T))
    return value
end

Reading 1.7 million UInt32s, with the results folded together so that the loop cannot vectorise away:

Reader ns per read
Base, IOBuffer 0.39
IOReader(CursorReader) 4.97
Direct load, CursorReader 1.34
IOReader(BufReader(IOBuffer)) 4.52
Direct load, BufReader(IOBuffer) 1.65

On a zstd stream the same change takes repeated read(s, UInt32) from 548 to 1489 MB/s, which is the speed of the decompression itself. For anyone parsing a binary format, the IO layer would stop being a cost at all.

I can imagine leaving typed reads off the abstract types was deliberate, to keep the interface small and push Base.IO behaviour into the wrappers. If so, the wrappers could take the same fast path instead, and a registered error hint on the MethodError pointing at IOReader would save newcomers a search. I have not prototyped the writing side.

3. Bulk reads do not go through read_into!

This one is a design question more than a patch, and I have not prototyped it inside BufferIO.

read_into!(x, dst) is documented as a method readers may specialise, and BufReader does. The generic read(x), readbytes!, and read! do not call it, though: they copy out of the reader's buffer. A reader that could produce bytes straight into the caller's memory therefore has to override each of those methods separately to avoid the intermediate copy.

ZstdLib is such a reader, as libzstd will decompress directly into whatever memory it is given. It currently overrides read(reader) to do that, and to size the result from the frame header when the frame records its content size. Against BufferIO's generic read on the same reader, the override is 22–45% faster on frames that record their size (1935 against 1488 MB/s on 8 MiB of JSON lines, 1748 against 1206 MB/s on 98 MiB), and allocates a third as much.

If the generic bulk reads were built on read_into!, a reader like this could implement that one method and get read, readbytes!, and read! for free. A second, optional hook for a reader to say how many bytes it expects to produce would let the generic read size its result up front, and I could then delete the override entirely.

Smaller things

  • line_views on a reader whose buffer cannot grow throws BufferTooShort when a line exceeds the buffer. That follows from returning views, but the message could suggest a larger buffer or eachline, both of which work.
  • VecWriter() itself costs 12.2 ns to construct, against 5.7 ns for Vector{UInt8}(undef, 32) and 3.2 ns for UInt8[]. Allocating its initial 32 bytes lazily would help any code that makes many small writers, readuntil aside.

I am happy to turn any of these into pull requests, or to split this into separate issues if that is easier to track. I would also be glad to hear if I have misread the intent of the interface anywhere, particularly in the third section.

Reproduction script (BufferIO and Chairmarks only)
using BufferIO, Chairmarks, Random
using BufferIO: MemoryView

# 1. `readuntil` copying straight from the buffer when it already holds the delimiter
function readuntil_direct(x::AbstractBufReader, delim::UInt8; keep::Bool = false)
    buffer = get_nonempty_buffer(x)
    buffer === nothing && return UInt8[]
    pos = findfirst(==(delim), buffer)
    pos === nothing && return readuntil(x, delim; keep)
    v = Vector{UInt8}(buffer[1:(pos - !keep)])
    consume(x, pos)
    return v
end

# 2. A typed read loading from the buffer when it holds the whole value
const Bits = Union{Int16, UInt16, Int32, UInt32, Int64, UInt64, Float32, Float64}
function read_direct(x::AbstractBufReader, ::Type{T}) where {T <: Bits}
    buffer = get_buffer(x)
    if length(buffer) < sizeof(T)
        bytes = Vector{UInt8}(undef, sizeof(T))
        read_all!(x, MemoryView(bytes)) == sizeof(T) || throw(EOFError())
        return only(reinterpret(T, bytes))
    end
    value = GC.@preserve buffer unsafe_load(Ptr{T}(pointer(buffer)))
    consume(x, sizeof(T))
    return value
end

rng = Xoshiro(1)
data = Vector{UInt8}(codeunits(join(("{\"id\": $i, \"user\": \"user$(rand(rng, 1:5000))\", \"score\": $(rand(rng, 1:1000)), \"tags\": [\"t$(i % 7)\", \"u$(i % 13)\"]}" for i in 1:100_000), '\n')))
resize!(data, length(data) & ~7)

# Both agree with what BufferIO does today, across buffer sizes that split values and fields
function fields(f, reader; kwargs...)
    out = Vector{UInt8}[]
    while !eof(reader); push!(out, f(reader, UInt8(','); kwargs...)); end
    out
end
values(f, reader, T) = [f(reader, T) for _ in 1:(length(data) ÷ sizeof(T))]
for bufsize in (1, 3, 7, 64, 4096), keep in (false, true)
    sample = data[1:20_000]
    @assert fields(readuntil_direct, BufReader(IOBuffer(sample), bufsize); keep) == fields(readuntil, BufReader(IOBuffer(sample), bufsize); keep)
end
for bufsize in (1, 3, 7, 64, 4096), T in (UInt16, UInt32, Int64, Float64)
    sample = data[1:20_000]
    direct = BufReader(IOBuffer(sample), bufsize); viaio = IOReader(BufReader(IOBuffer(sample), bufsize))
    @assert all(read_direct(direct, T) === read(viaio, T) for _ in 1:(length(sample) ÷ sizeof(T)))
end
println("prototypes agree with BufferIO ", pkgversion(BufferIO))

best(f) = minimum((@b f() seconds = 0.2).time for _ in 1:10)
function drain(f::F, reader) where {F}; n = 0; while !eof(reader); f(reader, UInt8(',')); n += 1; end; n; end
calls = drain(readuntil, CursorReader(data))
println("readuntil, ns per call over $calls calls:")
println("  Base, IOBuffer             ", round(best(() -> drain(readuntil, IOBuffer(data))) / calls * 1e9; digits = 1))
println("  BufferIO, CursorReader     ", round(best(() -> drain(readuntil, CursorReader(data))) / calls * 1e9; digits = 1))
println("  direct copy, CursorReader  ", round(best(() -> drain(readuntil_direct, CursorReader(data))) / calls * 1e9; digits = 1))
println("  BufferIO, BufReader        ", round(best(() -> drain(readuntil, BufReader(IOBuffer(data)))) / calls * 1e9; digits = 1))
println("  direct copy, BufReader     ", round(best(() -> drain(readuntil_direct, BufReader(IOBuffer(data)))) / calls * 1e9; digits = 1))
# xor-folding keeps the loop from vectorising away, so that each read is paid for
function fold(f::F, reader, ::Type{T}, n::Int) where {F, T}; acc = zero(T); for _ in 1:n; acc = bitrotate(acc, 1) ⊻ f(reader, T); end; acc; end
reads = length(data) ÷ 4
@assert fold(read_direct, BufReader(IOBuffer(data)), UInt32, reads) == fold(read, IOBuffer(data), UInt32, reads)
println("read(io, UInt32), ns per read over $reads reads:")
println("  Base, IOBuffer                  ", round(best(() -> fold(read, IOBuffer(data), UInt32, reads)) / reads * 1e9; digits = 2))
println("  IOReader(CursorReader)          ", round(best(() -> fold(read, IOReader(CursorReader(data)), UInt32, reads)) / reads * 1e9; digits = 2))
println("  direct load, CursorReader       ", round(best(() -> fold(read_direct, CursorReader(data), UInt32, reads)) / reads * 1e9; digits = 2))
println("  IOReader(BufReader(IOBuffer))   ", round(best(() -> fold(read, IOReader(BufReader(IOBuffer(data))), UInt32, reads)) / reads * 1e9; digits = 2))
println("  direct load, BufReader(IOBuffer)", round(best(() -> fold(read_direct, BufReader(IOBuffer(data)), UInt32, reads)) / reads * 1e9; digits = 2))

Activity

  1. changed the title [-]Feedback[/-] [+]Feedback from ZstdLib[/+] on Nov 13, 2026
  2. jakobnissen commented on Nov 13, 2026

    @jakobnissen
    Member

    Thanks for the issue. These points are now addressed, and are in version 0.3.3 which is undergoing registration:

    1. readuntil now has a fast path as suggested
    2. Several more bulk reads now go through read_into!. Surprisingly and happily, this change also sped up the generic implementation of these functions.
    3. VecWriter is now initialized without backing memory, making instantiation faster. Allocating the first memory is still slower than a Vector{UInt8} because VecWriter uses string-backed memory which requires another allocation. Maybe it shouldn't, but that's a discussion for another time.

    There are two things that I have intentionally not fixed.

    1. Typed reads and writes. I updated the docs to explain why BufferIO intentionally does not provide this. I expect to make a BufferSerde.jl package in the near future (i.e. some time the next year) to handle this.
    2. The throwing behaviour of line_views is well documented, and so are the workarounds. I would rather throw a well-typed error without a custom error message than a custom error message in a generic error.
  3. tecosaur commented on Nov 14, 2026

    @tecosaur
    Author

    Thanks for giving this a look and acting on it so quickly Jakob 🙂

    I got Claude to adjust ZstdLib to make use of the v0.3.3 changes: it now uses read_into! and that lets me get rid of most of the read override we had.

    There is one bit of Section 3 that you haven't commented on, and still motivates our own read method, that may have slipped by? There's no convenient way to say how many bytes a reader expects to produce, which allows for avoiding resizing, which with an eventual growth factor of 1.125x (overallocation_size) is far from free.

    Apologies for my laziness, but I'm once again going to paste comments by Claude below:


    The clearest case is BufferIO's own BufReader{IOStream}, which could answer with filesize(io) - position(io) plus whatever it has buffered, as Base does to size read(::IOStream). Reading a file from the page cache:

    File Base, read(::IOStream) read(BufReader(io)) today Presized read_all!
    8 MiB 11.9 GB/s 3.7 GB/s 17.4 GB/s
    100 MiB 6.2 GB/s 2.1 GB/s 6.4 GB/s

    Zstd frames that record their content size see the same effect: presizing reads 8 MiB of JSON lines at 2371 MB/s against 1704 MB/s for the generic read, and 100 MiB at 1951 against 1193 MB/s.

    What I have in mind is something like expected_remaining(x::AbstractBufReader)::Union{Nothing, Int}, defaulting to nothing, which the generic read consults whenever it needs to grow. Asking at each growth rather than once matters for concatenated frames, which each only reveal their size as they start. A wrong hint should only ever cost a resize, never correctness, so a reader can pass on a value it doesn't fully trust (ZstdLib clamps a frame's claim to 128 MiB).

  4. jakobnissen commented on Nov 14, 2026

    @jakobnissen
    Member

    Good catch, I missed that. #70 implements remaining_bytes_hint, which does this. Functions in BufferIO uses this method, which brings BufferIO read up to par with Base. I will cut a new release shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions