Skip to content

guide.md

Latest commit

 

History

History
184 lines (151 loc) · 9.44 KB

guide.md

File metadata and controls

184 lines (151 loc) · 9.44 KB
Raw

Developer Orientation Guide

Quick-reference guide to the chucky codebase: module layout, public API, internal conventions, and key concepts.

Module map

Platform & utilities

Target Source Purpose
dimension src/dimension.c struct dimension builder/validation helpers
chucky_log src/log/ Logging
platform src/platform/platform.*.c OS abstraction (page size, clock)
platform_io src/platform/platform_io.*.c Unbuffered file I/O
platform_cmd src/platform/platform_cmd.*.c Subprocess execution
io_scheduler src/zarr/io_scheduler.* Filesystem command ordering and execution
host_output_pool src/stream/host_output_pool.* Bounded host-output leases
index_ops src/util/index.ops.c Mixed-radix index arithmetic
crc32c src/zarr/crc32c.c CRC-32C checksum
writer src/writer.c struct writer dispatch + backpressure helpers

Data model

Header Purpose
dtype.h enum dtype (11 types) + dtype_bpe()
dimension.h struct dimension + builder helpers (dims_create, dims_set_chunk_sizes, etc.)
types.stream.h struct tile_stream_configuration, struct tile_stream_status, struct stream_metrics
types.codec.h struct codec_config: codec (none / lz4 / zstd / blosc-lz4 / blosc-zstd), level, shuffle, explicit Blosc block size
types.lod.h enum lod_reduce_method (mean / min / max / median / max_suppressed / min_suppressed)
defs.limits.h Compile-time limits: MAX_RANK=64, MAX_ZARR_RANK=32, LOD_MAX_LEVELS=32

Layout & planning

Target Source Purpose
stream_config src/stream/config.c Compute stream layouts, aggregate types, batch sizing
lod_plan src/lod/lod_plan.c Pure-C LOD plan: per-level shapes, counts, ends arrays

GPU backend

Target Source Purpose
transpose src/gpu/transpose.cu CUDA scatter kernel (input -> chunk pool)
compress src/gpu/compress.cu, src/gpu/blosc.* Raw LZ4/Zstd via nvCOMP; Blosc block preparation, compression, and framing
aggregate src/gpu/aggregate.cu Pack compressed chunks into shard buffers
lod src/gpu/lod.cu GPU LOD scatter + reduce (all 11 dtypes)
stream src/gpu/stream.c + helpers GPU pipeline orchestrator

CPU backend

Target Source Purpose
transpose_cpu src/cpu/transpose.cpp OpenMP scatter
compress_cpu src/cpu/compress.c, src/cpu/compress_blosc.c zstd / lz4 and optional c-blosc compression (CPU)
aggregate_cpu src/cpu/aggregate.c Shard packing (CPU)
lod_cpu src/cpu/lod.cpp CPU LOD scatter + reduce
stream_cpu src/cpu/stream.c CPU pipeline orchestrator

Zarr storage

Target Source Purpose
json_writer src/zarr/json_writer.c JSON serialization for zarr metadata
zarr_metadata src/zarr/zarr_metadata.c Serialize Zarr v3 array and group metadata
zarr_metadata_io src/zarr/metadata_io.c Construct metadata keys, submit snapshots, and wait for publication
zarr_array src/zarr/zarr_array.c Zarr array geometry, metadata, and shard sink
zarr_group src/zarr/zarr_group.c Zarr group envelopes and attributes
ngff_metadata src/ngff/ngff_metadata.c Serialize OME-NGFF multiscale attributes
ngff_multiscale src/ngff/ngff_multiscale.c Compose Zarr arrays and group metadata
shard_delivery src/zarr/shard_delivery.c Write shard index + CRC, deliver to shard_writer
shard_pool_fs src/zarr/shard_pool_fs.c Queue filesystem shard writes and metadata publication
io_backend_fs src/zarr/io_backend.fs.c Execute filesystem IO and own atomic metadata replacement
store_fs src/zarr/store_fs.c Filesystem store keys and pool creation
s3_client src/zarr/s3_client.c AWS S3 multipart upload client
shard_pool_s3 src/zarr/shard_pool_s3.c S3 shard upload slots
store_s3 src/zarr/store_s3.c S3 metadata keys and pool creation

Public API headers

A library consumer needs:

  • stream.gpu.h — GPU pipeline: create, destroy, writer, memory estimate
  • stream.cpu.h — CPU pipeline: same interface, no CUDA dependency
  • writer.h — struct writer, struct slice, writer_append(), writer_flush(), writer_close()
  • dimension.h — struct dimension + builder/validation helpers
  • dtype.h — enum dtype, dtype_bpe()
  • types.stream.h — struct tile_stream_configuration, metrics, status
  • types.codec.h — enum compression_codec, enum codec_shuffle, struct codec_config
  • types.lod.h — enum lod_reduce_method
  • defs.limits.h — compile-time limits
  • chucky_log.h — log level / callback control for routing chucky's log output
  • zarr_fs_sink.h — filesystem Zarr sink
  • zarr_s3_sink.h — S3 Zarr sink

Internal conventions

Error handling. Functions return int (0 = success, non-zero = error). Callers test with if (func()). Macros: CHECK(label, expr) for assertions with goto-cleanup, CU(label, expr) for CUDA calls.

Naming. Dots for namespacing headers (stream.gpu.h, types.stream.h). Underscores within identifiers (platform_io, lod_plan). Prefix tile_stream_ for the public streaming API.

Memory. buffer_new / buffer_free for pinned host or device memory with CUevent synchronization. Host buffers use CU_MEMHOSTALLOC_WRITECOMBINED — do not read from the host side; copy data out first.

Writer vtable. struct writer has append(self, slice), flush(self) and an optional close(self), each returning a writer_result. Free functions writer_append() / writer_flush() / writer_close() dispatch through it. flush finalizes: it writes out everything appended so far and then stops taking input, so it is the last call on a stream rather than a mid-stream sync. It returns once those writes have landed. writer_close() then publishes the extent and reports whether the array is readable.

Two-phase init. compute_stream_layouts() does all pure-CPU layout math (lifted shape, strides, chunk geometry), then the GPU path uploads to device memory. tile_stream_gpu_memory_estimate reuses the same layout function.

GPU vs CPU backends

Both backends implement the same pipeline stages behind the same struct writer interface:

Stage GPU CPU
Scatter CUDA kernel OpenMP parallel loop
Compress nvCOMP LZ4/Zstd, raw or Blosc-framed libzstd / liblz4 / optional c-blosc
Aggregate CUDA kernel Sequential packing
LOD CUDA kernels (templated on dtype) C++ templates + OpenMP
Orchestration 4 CUDA streams, double-buffered Single-threaded pipeline

Create with tile_stream_gpu_create() or tile_stream_cpu_create(). Both return a struct writer* via _writer() — downstream code is identical.

Both Blosc codecs support no shuffle, byte shuffle, and bitshuffle. Set codec.blosc_block_bytes explicitly on both backends, including at level 0; zero is invalid. GPU Blosc uses internal blocks capped by the Zarr chunk size, while CPU C-Blosc may adjust the requested size. See the configuration examples for limits, level semantics, and matching the stream and sink settings.

Key concepts

  • epoch — one full pass of the inner chunk dimensions; the append dimension advances by chunk_size[0] per epoch
  • batch — K epochs grouped for compression; K is auto-tuned or set via epochs_per_batch
  • chunk — Zarr's independently compressed unit; shaped by dimension.chunk_size per axis
  • Blosc block — an internal portion of a chunk, filtered and compressed independently; changing its size does not change Zarr chunk or shard geometry
  • shard — a file containing multiple compressed chunks plus a binary index
  • lifted shape — the input tensor reshaped as (t[D-1], n[D-1], ..., t[0], n[0]) to expose chunk structure
  • append dimension — dimension 0; may have size=0 (unbounded, streams indefinitely)
  • inner dimensions — dimensions 1..rank-1; fully known at stream creation
  • LOD level — one layer of the multiscale pyramid; L0 = full resolution
  • compacted morton order — Z-order curve over chunk indices within a shard, used to map LOD chunks to shard positions

Further reading