VeloxRssSortShuffleReaderDeserializer: infinite loop on EOS mid-page and corrupted deserialization on multi-window Presto pages

[kecookier]

Backend

VL (Velox)

Bug description

Description

We hit two bugs in the rss_sort shuffle reader (VeloxRssSortShuffleReaderDeserializer::VeloxInputStream) in our production environment with Celeborn (rss_sort + hash partitioning). Both reproduce on current main — regression tests submitted in #12983 fail.

Bug 1: infinite loop on EOS mid-page

GlutenByteInputStream::readBytes() drives next(true) in a for(;;) loop. When a page header declares more bytes than the stream actually holds (truncated partition data), the reader hits EOS mid-page — but VeloxInputStream::next() ignores its throwIfPastEnd argument and silently returns on EOS, so the loop never exits.

Symptom in production: the task appears stuck (thread in RUNNABLE, no progress, no error surfaced).

CI evidence from #12983:

C++ exception with description "possible infinite loop: Read() returned 0 for 100 consecutive calls" thrown in the test body.

(the test's fake stream cuts the loop short after 100 consecutive EOS reads instead of hanging)

Additionally, next() stores the Read() result into an unsigned offset_ without a signed guard, so a negative return from a buggy upstream corrupts setRange instead of failing fast.

Bug 2: corrupted deserialization when a page spans multiple read windows

VeloxInputStream is a single-window refill stream: each next() overwrites the sole ByteRange with serde assumes stable multi-range data — for nested types it does tellp() → pre-scan the page →seekp() back, and it scans the payload via nextView() for checksum verification. When an uncompressed page (nested structs) spans multiple ~1MB read windows, the serde's backward seek reads window data already overwritten by a
refill, corrupting the stream.

CI evidence from #12983:

Exception: VeloxRuntimeError
Reason: (-673841146 vs. 0) Invalid serialized string size: -673841146
Function: readLengthPrefixedString
# 3 ...readLengthPrefixedString(ByteInputStream*)
# 6 ...PrestoVectorSerde::deserialize(...)
# 9 gluten::VeloxRssSortShuffleReaderDeserializer::next()

Impact

• Bug 1: task hangs forever with no error — requires external timeout to kill, hard to diagnose from logs.
• Bug 2: wrong data / spurious deserialization errors for pages larger than one read window (~1MB) containing nested types.

Fix

We have a fix ready and will submit a PR shortly (referencing this issue).

Gluten version

main branch, Gluten-1.3

Spark version

Spark-3.5.x

Spark configurations

No response

System information

No response

Relevant logs

[kecookier]

@ccat3z