Repository navigation
Description
Hey Jakob,
As you know, I've been working with Claude on ZstdLib, and BufferIO has worked pretty well overall (6x the line iteration speed vs. TranscodingStreams is a nice headline result, and it's trim-compatible!).
That said, a few slightly less smooth spots turned up, and so I've lazily asked for them to be written up into an issue, which I've just pasted below:
🤖 Fable 5.2
Faster readuntil, typed reads, and bulk reads on AbstractBufReader
While benchmarking I came across three places where the generic reader methods leave some speed on the table, and thought the measurements might be useful to you.
All numbers are from BufferIO 0.3.2 on Julia 1.13.2 (i5-13600K, one pinned core), taking the fastest of ten 0.2 s runs. The script at the end reproduces the first two sections using BufferIO alone, and checks that both prototypes return exactly what BufferIO returns today across buffer sizes from 1 to 4096 bytes.
1. readuntil builds a VecWriter on every call
readuntil(x, delim) creates a VecWriter and hands it to copyuntil. Constructing the writer takes about 12 ns (it allocates a 32-byte string-backed vector), against 6 ns for Vector{UInt8}(undef, 32), and the bytes then go through the general write path.
When the delimiter is already in the buffer, which it is for most short fields, the bytes can be copied out directly. That is the same shape as the fast path readline already has:
function Base.readuntil(x::AbstractBufReader, delim::UInt8; keep::Bool = false)
buffer = get_nonempty_buffer(x)::Union{Nothing, ImmutableMemoryView{UInt8}}
buffer === nothing && return UInt8[]
pos = findfirst(==(delim), buffer)
pos === nothing && return readuntil_slowpath(x, delim, keep) # today's implementation
v = Vector{UInt8}(@inbounds MemoryViews.truncate(buffer, pos - !keep))
@inbounds consume(x, pos)
return v
endSplitting 8 MiB of JSON lines on commas (400,001 calls):
| Reader | ns per call |
|---|---|
Base, IOBuffer |
37.2 |
BufferIO today, CursorReader |
20.9 |
Direct copy, CursorReader |
12.8 |
BufferIO today, BufReader(IOBuffer) |
21.0 |
Direct copy, BufReader(IOBuffer) |
12.7 |
BufferIO is already well ahead of Base here, so this is an improvement on something good. Downstream it matters a little more than the table suggests: on a zstd stream it takes readuntil from 532 to 846 MB/s, which moves it from behind CodecZstd (739 MB/s) to ahead of it.
2. Typed reads on the abstract types
read(reader, UInt32) on an AbstractBufReader is a MethodError, and so is write(writer, 0x01020304) on an AbstractBufWriter. Both work through IOReader and IOWriter, though that route goes through Base's generic machinery. Since the reader already exposes its buffer, a value that lies wholly within it can be loaded directly:
function Base.read(x::AbstractBufReader, ::Type{T}) where {T <: Union{Int16, UInt16, Int32, UInt32, Int64, UInt64, Float32, Float64}}
buffer = get_buffer(x)::ImmutableMemoryView{UInt8}
length(buffer) < sizeof(T) && return read_slowpath(x, T) # refill, or copy across a buffer boundary
value = GC.@preserve buffer unsafe_load(Ptr{T}(pointer(buffer)))
@inbounds consume(x, sizeof(T))
return value
endReading 1.7 million UInt32s, with the results folded together so that the loop cannot vectorise away:
| Reader | ns per read |
|---|---|
Base, IOBuffer |
0.39 |
IOReader(CursorReader) |
4.97 |
Direct load, CursorReader |
1.34 |
IOReader(BufReader(IOBuffer)) |
4.52 |
Direct load, BufReader(IOBuffer) |
1.65 |
On a zstd stream the same change takes repeated read(s, UInt32) from 548 to 1489 MB/s, which is the speed of the decompression itself. For anyone parsing a binary format, the IO layer would stop being a cost at all.
I can imagine leaving typed reads off the abstract types was deliberate, to keep the interface small and push Base.IO behaviour into the wrappers. If so, the wrappers could take the same fast path instead, and a registered error hint on the MethodError pointing at IOReader would save newcomers a search. I have not prototyped the writing side.
3. Bulk reads do not go through read_into!
This one is a design question more than a patch, and I have not prototyped it inside BufferIO.
read_into!(x, dst) is documented as a method readers may specialise, and BufReader does. The generic read(x), readbytes!, and read! do not call it, though: they copy out of the reader's buffer. A reader that could produce bytes straight into the caller's memory therefore has to override each of those methods separately to avoid the intermediate copy.
ZstdLib is such a reader, as libzstd will decompress directly into whatever memory it is given. It currently overrides read(reader) to do that, and to size the result from the frame header when the frame records its content size. Against BufferIO's generic read on the same reader, the override is 22–45% faster on frames that record their size (1935 against 1488 MB/s on 8 MiB of JSON lines, 1748 against 1206 MB/s on 98 MiB), and allocates a third as much.
If the generic bulk reads were built on read_into!, a reader like this could implement that one method and get read, readbytes!, and read! for free. A second, optional hook for a reader to say how many bytes it expects to produce would let the generic read size its result up front, and I could then delete the override entirely.
Smaller things
line_viewson a reader whose buffer cannot grow throwsBufferTooShortwhen a line exceeds the buffer. That follows from returning views, but the message could suggest a larger buffer oreachline, both of which work.VecWriter()itself costs 12.2 ns to construct, against 5.7 ns forVector{UInt8}(undef, 32)and 3.2 ns forUInt8[]. Allocating its initial 32 bytes lazily would help any code that makes many small writers,readuntilaside.
I am happy to turn any of these into pull requests, or to split this into separate issues if that is easier to track. I would also be glad to hear if I have misread the intent of the interface anywhere, particularly in the third section.
Reproduction script (BufferIO and Chairmarks only)
using BufferIO, Chairmarks, Random
using BufferIO: MemoryView
# 1. `readuntil` copying straight from the buffer when it already holds the delimiter
function readuntil_direct(x::AbstractBufReader, delim::UInt8; keep::Bool = false)
buffer = get_nonempty_buffer(x)
buffer === nothing && return UInt8[]
pos = findfirst(==(delim), buffer)
pos === nothing && return readuntil(x, delim; keep)
v = Vector{UInt8}(buffer[1:(pos - !keep)])
consume(x, pos)
return v
end
# 2. A typed read loading from the buffer when it holds the whole value
const Bits = Union{Int16, UInt16, Int32, UInt32, Int64, UInt64, Float32, Float64}
function read_direct(x::AbstractBufReader, ::Type{T}) where {T <: Bits}
buffer = get_buffer(x)
if length(buffer) < sizeof(T)
bytes = Vector{UInt8}(undef, sizeof(T))
read_all!(x, MemoryView(bytes)) == sizeof(T) || throw(EOFError())
return only(reinterpret(T, bytes))
end
value = GC.@preserve buffer unsafe_load(Ptr{T}(pointer(buffer)))
consume(x, sizeof(T))
return value
end
rng = Xoshiro(1)
data = Vector{UInt8}(codeunits(join(("{\"id\": $i, \"user\": \"user$(rand(rng, 1:5000))\", \"score\": $(rand(rng, 1:1000)), \"tags\": [\"t$(i % 7)\", \"u$(i % 13)\"]}" for i in 1:100_000), '\n')))
resize!(data, length(data) & ~7)
# Both agree with what BufferIO does today, across buffer sizes that split values and fields
function fields(f, reader; kwargs...)
out = Vector{UInt8}[]
while !eof(reader); push!(out, f(reader, UInt8(','); kwargs...)); end
out
end
values(f, reader, T) = [f(reader, T) for _ in 1:(length(data) ÷ sizeof(T))]
for bufsize in (1, 3, 7, 64, 4096), keep in (false, true)
sample = data[1:20_000]
@assert fields(readuntil_direct, BufReader(IOBuffer(sample), bufsize); keep) == fields(readuntil, BufReader(IOBuffer(sample), bufsize); keep)
end
for bufsize in (1, 3, 7, 64, 4096), T in (UInt16, UInt32, Int64, Float64)
sample = data[1:20_000]
direct = BufReader(IOBuffer(sample), bufsize); viaio = IOReader(BufReader(IOBuffer(sample), bufsize))
@assert all(read_direct(direct, T) === read(viaio, T) for _ in 1:(length(sample) ÷ sizeof(T)))
end
println("prototypes agree with BufferIO ", pkgversion(BufferIO))
best(f) = minimum((@b f() seconds = 0.2).time for _ in 1:10)
function drain(f::F, reader) where {F}; n = 0; while !eof(reader); f(reader, UInt8(',')); n += 1; end; n; end
calls = drain(readuntil, CursorReader(data))
println("readuntil, ns per call over $calls calls:")
println(" Base, IOBuffer ", round(best(() -> drain(readuntil, IOBuffer(data))) / calls * 1e9; digits = 1))
println(" BufferIO, CursorReader ", round(best(() -> drain(readuntil, CursorReader(data))) / calls * 1e9; digits = 1))
println(" direct copy, CursorReader ", round(best(() -> drain(readuntil_direct, CursorReader(data))) / calls * 1e9; digits = 1))
println(" BufferIO, BufReader ", round(best(() -> drain(readuntil, BufReader(IOBuffer(data)))) / calls * 1e9; digits = 1))
println(" direct copy, BufReader ", round(best(() -> drain(readuntil_direct, BufReader(IOBuffer(data)))) / calls * 1e9; digits = 1))
# xor-folding keeps the loop from vectorising away, so that each read is paid for
function fold(f::F, reader, ::Type{T}, n::Int) where {F, T}; acc = zero(T); for _ in 1:n; acc = bitrotate(acc, 1) ⊻ f(reader, T); end; acc; end
reads = length(data) ÷ 4
@assert fold(read_direct, BufReader(IOBuffer(data)), UInt32, reads) == fold(read, IOBuffer(data), UInt32, reads)
println("read(io, UInt32), ns per read over $reads reads:")
println(" Base, IOBuffer ", round(best(() -> fold(read, IOBuffer(data), UInt32, reads)) / reads * 1e9; digits = 2))
println(" IOReader(CursorReader) ", round(best(() -> fold(read, IOReader(CursorReader(data)), UInt32, reads)) / reads * 1e9; digits = 2))
println(" direct load, CursorReader ", round(best(() -> fold(read_direct, CursorReader(data), UInt32, reads)) / reads * 1e9; digits = 2))
println(" IOReader(BufReader(IOBuffer)) ", round(best(() -> fold(read, IOReader(BufReader(IOBuffer(data))), UInt32, reads)) / reads * 1e9; digits = 2))
println(" direct load, BufReader(IOBuffer)", round(best(() -> fold(read_direct, BufReader(IOBuffer(data)), UInt32, reads)) / reads * 1e9; digits = 2))
Activity
[-]Feedback[/-][+]Feedback from ZstdLib[/+]jakobnissen commentedon Nov 13, 2026
Thanks for the issue. These points are now addressed, and are in version 0.3.3 which is undergoing registration:
readuntilnow has a fast path as suggestedread_into!. Surprisingly and happily, this change also sped up the generic implementation of these functions.VecWriteris now initialized without backing memory, making instantiation faster. Allocating the first memory is still slower than aVector{UInt8}becauseVecWriteruses string-backed memory which requires another allocation. Maybe it shouldn't, but that's a discussion for another time.There are two things that I have intentionally not fixed.
line_viewsis well documented, and so are the workarounds. I would rather throw a well-typed error without a custom error message than a custom error message in a generic error.tecosaur commentedon Nov 14, 2026
Thanks for giving this a look and acting on it so quickly Jakob 🙂
I got Claude to adjust ZstdLib to make use of the v0.3.3 changes: it now uses
read_into!and that lets me get rid of most of thereadoverride we had.There is one bit of Section 3 that you haven't commented on, and still motivates our own
readmethod, that may have slipped by? There's no convenient way to say how many bytes a reader expects to produce, which allows for avoiding resizing, which with an eventual growth factor of 1.125x (overallocation_size) is far from free.Apologies for my laziness, but I'm once again going to paste comments by Claude below:
The clearest case is BufferIO's own
BufReader{IOStream}, which could answer withfilesize(io) - position(io)plus whatever it has buffered, asBasedoes to sizeread(::IOStream). Reading a file from the page cache:Base,read(::IOStream)read(BufReader(io))todayread_all!Zstd frames that record their content size see the same effect: presizing reads 8 MiB of JSON lines at 2371 MB/s against 1704 MB/s for the generic
read, and 100 MiB at 1951 against 1193 MB/s.What I have in mind is something like
expected_remaining(x::AbstractBufReader)::Union{Nothing, Int}, defaulting tonothing, which the genericreadconsults whenever it needs to grow. Asking at each growth rather than once matters for concatenated frames, which each only reveal their size as they start. A wrong hint should only ever cost a resize, never correctness, so a reader can pass on a value it doesn't fully trust (ZstdLib clamps a frame's claim to 128 MiB).jakobnissen commentedon Nov 14, 2026
Good catch, I missed that. #70 implements
remaining_bytes_hint, which does this. Functions in BufferIO uses this method, which brings BufferIOreadup to par with Base. I will cut a new release shortly.