Local storage range downloads buffer the source and ignore chunk_size #2191

Summary

Local storage range downloads allocate memory proportional to the entire source file, even when only a small range is requested, and ignore chunk_size. The file-download path also buffers the complete requested range before writing it.

Detailed Information

Reproduced on trunk at 8d659f527f28c76db41f19c86d056a2338717b8c, Python 3.12.14 on Linux.

import os
import tempfile

from libcloud.storage.drivers.local import LocalStorageDriver

with tempfile.TemporaryDirectory() as root:
    driver = LocalStorageDriver(root)
    container = driver.create_container("example")
    with open(os.path.join(root, "example", "object"), "wb") as fp:
        fp.write(b"0123456789" * 2000)
    obj = container.get_object("object")
    chunks = list(driver.download_object_range_as_stream(
        obj, start_bytes=5, end_bytes=12, chunk_size=3
    ))
    print(chunks)

Actual: [b'5678901'], following an unbounded read of the entire 20,000-byte source file.
Expected: [b'567', b'890', b'1'], reading only the seven requested bytes.

download_object_range_as_stream calls obj_file.read() without a size to determine file length, then reads the entire requested range into one chunk. download_object_range calls exhaust_iterator before writing anything.

A local tracemalloc experiment requesting 1 KiB from a 64 MiB sparse file with chunk_size=128 measured 67,113,884 bytes of peak traced allocation before the fix and 6,242 bytes afterward. This measures Python allocations for that experiment, not total process memory.

A focused fix and regression tests are prepared: use descriptor metadata for file size, read only the selected range in bounded chunks, and write chunks as they arrive. Existing non-inclusive range boundaries are preserved.

AI assistance: This report and the proposed fix were prepared with OpenAI Codex (GPT-6).