diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a1598d3..1bc99163 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,134 @@ Format follows [Keep a Changelog](https://keepachangelog.com/). ## [Unreleased] +### Added — structural introspection of a message in every binding + +A message's *structure* can now be read without decoding a payload or a byte +of CBOR, in **Rust, C, C++, Python, TypeScript, and Fortran**. Two entry +points, both taking the bytes of one message: a **message header** that +decodes the 24-byte preamble into typed values (wire version, `total_length`, +and eight `has_*` predicates naming the optional frames — so a random-access +layout is distinguishable from a streaming one after 24 bytes), and a **frame +walk** that reports each frame's type, frame-header version and flags, its +`offset` / `length` within the message, its content bytes, and whether its +inline hash slot holds a real digest. The preamble and postamble are *not* +frames and are never yielded; inter-frame alignment padding belongs to no +frame. `payload` is the frame **content**, with the 16-byte frame header and +the type-specific footer (20 bytes for the data-object type, 12 otherwise) +stripped. + +- **Core (Rust)** — `tensogram::frames(message)`, a lazy `FrameIter` yielding + `FrameInfo { frame_type, version, flags, offset, length, payload }` plus + `has_hash()` (one `Err` item on a malformed chain, then it stops), and + `tensogram::message_header(message) -> MessageHeader`, both in the new + `frame_walk` module. Every binding mirrors this one. +- **C** — the `tgm_frame_type` enum, the `TgmFrame` / `TgmMessageHeader` PODs, + `tgm_frame_iter_create` / `_next` / `_free`, `tgm_frame_has_hash`, and + `tgm_message_header`. `TgmFrame::payload` borrows the caller's message + buffer: nothing to free, valid for as long as that buffer lives, and + unaffected by later `_next` calls or by freeing the cursor. +- **C++** — `enum class frame_type`, a `frame` value type, a lazy move-only + `frame_range` usable in a range-for, `frames()`, and + `read_message_header()`. Payload views borrow the buffer like the C ones; a + malformed chain throws `framing_error` from the increment that finds it, + after the intact frames have been yielded. +- **Python** — `frames(buf) -> FrameIter` yielding `Frame` (`.frame_type` + name, `.frame_type_code`, `.version`, `.flags`, `.offset`, `.length`, + `.payload`, `.has_hash`, `.is_data_object`) and `message_header(buf) -> + MessageHeader`. +- **TypeScript** — `frames(buf) -> Frame[]` and `messageHeader(buf) -> + MessageHeader`. The array is eager, so a malformed chain throws rather than + silently returning fewer frames. +- **Fortran** — `tensogram_frames()` returning a non-copyable + `tensogram_frame_iterator` (`%next`), the `tensogram_frame` value type + (`%frame_type()`, `%offset()`, `%length()`, `%payload()`, `%has_hash()`), + and `tensogram_message_header_read()`. + +Python, TypeScript, and Fortran **copy** the payload bytes (Fortran also +copies the whole message into the iterator), so frames there outlive both the +walk and the caller's buffer. Both entry points describe exactly one message — +use `scan()` to locate boundaries in a multi-message buffer and slice — and +frame offsets are relative to that message (**1-based in Fortran**, 0-based +elsewhere). One documented asymmetry: in *streaming* mode the encoder writes +the preamble before the first object exists, so its flags only guarantee +"frame present ⇒ flag set" (`PRECEDER_METADATA` is set advisorily) and +`total_length` may stay `0`; for a buffered message the flags are exact. The +contract is documented in the new *Frame Introspection* guide +(`docs/src/guide/frame-introspection.md`), with runnable examples for C++, +Python, TypeScript, and Fortran. + +### Added — synchronous remote access in C, C++, and Fortran + +Opening a remote `.tgm` (S3, GCS, Azure, HTTP) previously required the async C +API. It is now available through the ordinary **blocking** file API: +`tgm_file_open_remote(source, keys[], values[], n, opts, out)` returns a plain +`tgm_file_t`, so the whole existing file surface — message count, raw read, +decode, iterators — works unchanged against a remote source, and +`tgm_file_close` closes it. Backend storage options (credentials, region, +endpoint, …) are passed as parallel key / value arrays, and +`TgmRemoteScanOptions { bidirectional }` configures the scan walker. +`tgm_is_remote_url` tells the remote and local backends apart before opening. +C++ adds `tensogram::is_remote_url` and +`file::open_remote(source, storage_options, opts)`; Fortran adds +`tensogram_is_remote_url` and `tensogram_file_open_remote` (with and without +option arrays). + +This is an **opt-in `remote` Cargo feature** on `tensogram-ffi`, independent of +`async`, and it is **not** part of the default published C-API tarballs — same +model as the existing `async` feature. Build it with +`cargo cinstall -p tensogram-ffi --features=remote`, or +`cmake -S cpp -B build -DTENSOGRAM_REMOTE=ON` (default OFF) for the C++ +wrapper. Both symbols are exported either way, so consumers never hit an +undefined symbol: a build without the feature answers `false` from +`tgm_is_remote_url` for every input and `TGM_ERROR_REMOTE` from +`tgm_file_open_remote`, with a message naming `--features=remote`. Argument +validation runs *before* the feature check, so a genuine mistake is reported as +`TGM_ERROR_INVALID_ARG` in either build. + +### Added — typed enums and the full encode-option set + +The C ABI gained four enums whose **zero value is the library default**, so a +zero-initialised option struct requests exactly the previous behaviour: +`tgm_dtype` (15 variants), `tgm_byte_order`, `tgm_aggregate_hash_policy` +(auto / none / header / footer / both), and `tgm_compression_backend` +(auto / ffi / pure). Two typed accessors — +`tgm_object_dtype_enum` / `tgm_object_byte_order_enum` — join the existing +string getters, which are unchanged and still supported; because an enum +return has no spare code for failure, a bad index yields the zero variant plus +a `tgm_last_error()` reason (the paired string getter returns NULL for exactly +the same inputs). The new `TgmEncodeOptions` POD supersedes +`TgmEncodeMaskOptions` (retained) by carrying the same six mask fields **plus** +the hash algorithm, the aggregate-hash placement, and the codec backend, with +one `*_with_encode_options` entry point per encode target (`NULL` ⇒ defaults). + +- **C++** — `enum class dtype` / `byte_order` / `aggregate_hash_policy` / + `compression_backend`, `decoded_object::dtype_enum()` / + `byte_order_enum()` (which raise on a bad index instead of returning a + plausible zero variant), and `encode_options::aggregate_hash` / + `::codec_backend`. Deliberately **no** `*_with_encode_options` overloads — + they would be signature-identical — so the existing entry points escalate to + the full C function automatically when either field is non-default; a caller + who sets neither keeps the exact previous call path and bytes. + `encode_pre_encoded()` has no full-option entry point in the C ABI and + therefore rejects both fields rather than ignoring them. +- **Fortran** — `tensogram_object_dtype_enum` / `_byte_order_enum` (with an + optional `err`), the `TGM_DTYPE_*` / `TGM_BYTE_ORDER_*` / + `TGM_AGGREGATE_HASH_POLICY_*` / `TGM_COMPRESSION_BACKEND_*` / + `TGM_FRAME_TYPE_*` parameters, and `tensogram_encode_with_options` / + `tensogram_file_append_with_options` / + `tensogram_streaming_encoder_create_with_options`. +- **Python** — `compression_backend=` on `encode`, `TensogramFile.append`, + `StreamingEncoder`, and `AsyncStreamingEncoder.create` (`aggregate_hash=` + already existed). +- **TypeScript** — `aggregateHash` on `encode()` and `TensogramFile#append`. + `compressionBackend` is accepted and validated there but is a documented + **no-op on WASM**, which ships the pure-Rust codecs only; it exists for + source symmetry with the other bindings. + +Header-side aggregate-hash placements (`HEADER`, `BOTH`) are buffered-mode +only: every streaming constructor rejects them, because a streaming writer +emits its header before any data object exists. + ## [0.24.0] - 2026-07-23 ### Added — symmetric metadata access across every binding diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index 7ffaee74..8cef0b7d 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -19,10 +19,33 @@ if(TENSOGRAM_ASYNC_REMOTE AND NOT TENSOGRAM_ASYNC) message(STATUS "TENSOGRAM_ASYNC_REMOTE=ON -> forcing TENSOGRAM_ASYNC=ON") set(TENSOGRAM_ASYNC ON) endif() + +# --------------------------------------------------------------------------- +# Synchronous remote surface +# --------------------------------------------------------------------------- +# TENSOGRAM_REMOTE builds the FFI with --features=remote, which backs +# tensogram::is_remote_url and tensogram::file::open_remote with a real +# object-store client. Deliberately independent of the async options: a +# caller reading an S3 / GCS / Azure / HTTP `.tgm` through the ordinary +# blocking file API should not have to pull in the async runtime. The FFI +# symbols always link, so with this OFF (the default, keeping the common +# build lean) those entry points still exist and report that this build +# cannot open a remote source. +option(TENSOGRAM_REMOTE + "Build the synchronous object-store backend (tgm_file_open_remote)" OFF) + +set(CARGO_FEATURE_LIST "") if(TENSOGRAM_ASYNC_REMOTE) - set(CARGO_FEATURES "--features=async-remote") + list(APPEND CARGO_FEATURE_LIST "async-remote") elseif(TENSOGRAM_ASYNC) - set(CARGO_FEATURES "--features=async") + list(APPEND CARGO_FEATURE_LIST "async") +endif() +if(TENSOGRAM_REMOTE) + list(APPEND CARGO_FEATURE_LIST "remote") +endif() +if(CARGO_FEATURE_LIST) + list(JOIN CARGO_FEATURE_LIST "," CARGO_FEATURE_CSV) + set(CARGO_FEATURES "--features=${CARGO_FEATURE_CSV}") else() set(CARGO_FEATURES "") endif() @@ -80,6 +103,9 @@ endif() if(TENSOGRAM_ASYNC_REMOTE) target_compile_definitions(tensogram INTERFACE TENSOGRAM_ASYNC_REMOTE=1) endif() +if(TENSOGRAM_REMOTE) + target_compile_definitions(tensogram INTERFACE TENSOGRAM_REMOTE=1) +endif() # Platform-specific system libraries required by the Rust static library if(APPLE) diff --git a/cpp/include/tensogram.hpp b/cpp/include/tensogram.hpp index 2556c482..ff18a33e 100644 --- a/cpp/include/tensogram.hpp +++ b/cpp/include/tensogram.hpp @@ -25,6 +25,11 @@ /// (data(), data_as(), shape(), strides(), etc.) borrow from the /// parent message handle and are valid only until that handle is /// destroyed or moved-from. +/// +/// @note Lifetime: the frame walker borrows differently — frames() views the +/// caller's message buffer directly, so frame::payload() stays valid +/// for as long as *that buffer* lives, outliving the frame_range that +/// produced it. See frame_range for the full contract. #ifndef TENSOGRAM_HPP #define TENSOGRAM_HPP @@ -32,6 +37,7 @@ #include #include #include +#include #include #include #include @@ -325,6 +331,172 @@ namespace detail { } // namespace detail +/// An element type, mirroring the C `tgm_dtype` enum and +/// `tensogram::Dtype` one-for-one. +/// +/// The wire format stores the dtype as a **string** (see +/// `plans/WIRE_FORMAT.md` §6.1), so these codes are a binding convenience +/// rather than a wire value — but they are part of the frozen C ABI. +/// decoded_object::dtype_string() returns the same information as text; +/// this enum is what you want in a `switch`. +enum class dtype { + float16 = TGM_DTYPE_FLOAT16, + bfloat16 = TGM_DTYPE_BFLOAT16, + float32 = TGM_DTYPE_FLOAT32, + float64 = TGM_DTYPE_FLOAT64, + complex64 = TGM_DTYPE_COMPLEX64, + complex128 = TGM_DTYPE_COMPLEX128, + int8 = TGM_DTYPE_INT8, + int16 = TGM_DTYPE_INT16, + int32 = TGM_DTYPE_INT32, + int64 = TGM_DTYPE_INT64, + uint8 = TGM_DTYPE_UINT8, + uint16 = TGM_DTYPE_UINT16, + uint32 = TGM_DTYPE_UINT32, + uint64 = TGM_DTYPE_UINT64, + /// Sub-byte packed bitmask; the element width is 1 bit, so the byte + /// width of this dtype is reported as 0. + bitmask = TGM_DTYPE_BITMASK, +}; + +/// A payload's byte order, mirroring the C `tgm_byte_order` enum and +/// `tensogram::ByteOrder`. +/// +/// Like dtype these codes are a binding convenience — the wire stores +/// `"little"` / `"big"` as text — and are frozen as part of the C ABI. +/// decoded_object::byte_order_string() returns the same information as text. +enum class byte_order { + little = TGM_BYTE_ORDER_LITTLE, + big = TGM_BYTE_ORDER_BIG, +}; + +/// Where to place the aggregate hash frame, mirroring the C +/// `tgm_aggregate_hash_policy` enum and `tensogram::AggregateHashPolicy`. +/// +/// The default is aggregate_hash_policy::automatic, so a +/// default-constructed encode_options asks the encoder to choose — header +/// when buffering, footer when streaming — exactly as before this knob +/// existed. +/// +/// `header` and `both` are **buffered-mode only**: a streaming encoder +/// writes its header before any data object exists, so the per-object +/// hashes are not yet known. Passing either to streaming_encoder throws +/// tensogram::encoding_error. +enum class aggregate_hash_policy { + /// Encoder picks: buffered → header, streaming → footer. + automatic = TGM_AGGREGATE_HASH_POLICY_AUTO, + /// Emit no aggregate hash frame. Per-frame inline hash slots are + /// unaffected — see frame::has_hash(). + none = TGM_AGGREGATE_HASH_POLICY_NONE, + /// Emit a frame_type::header_hash frame. Buffered mode only. + header = TGM_AGGREGATE_HASH_POLICY_HEADER, + /// Emit a frame_type::footer_hash frame. Valid in both modes. + footer = TGM_AGGREGATE_HASH_POLICY_FOOTER, + /// Emit both a header and a footer hash frame carrying identical hash + /// lists. Buffered mode only. + both = TGM_AGGREGATE_HASH_POLICY_BOTH, +}; + +/// Which codec implementation to use where both are compiled in, mirroring +/// the C `tgm_compression_backend` enum and `tensogram::CompressionBackend`. +/// +/// `automatic` (the default) consults the `TENSOGRAM_COMPRESSION_BACKEND` +/// environment variable, then the platform default. `ffi` and `pure` are +/// explicit overrides that always win over the environment. The choice +/// affects szip and zstd, the two codecs with both a C and a pure-Rust +/// implementation; the decoded payload is identical either way, only the +/// compressed bytes may differ. +enum class compression_backend { + /// Consult the environment, then the platform default. + automatic = TGM_COMPRESSION_BACKEND_AUTO, + /// Use the C FFI codecs (libaec for szip, libzstd for zstd). + ffi = TGM_COMPRESSION_BACKEND_FFI, + /// Use the pure-Rust codecs (tensogram-szip, ruzstd). + pure = TGM_COMPRESSION_BACKEND_PURE, +}; + +namespace detail { + +/// Map a C `tgm_dtype` discriminant to the C++ dtype enum. +[[nodiscard]] inline dtype dtype_from_c(tgm_dtype t) noexcept { + switch (t) { + case TGM_DTYPE_FLOAT16: return dtype::float16; + case TGM_DTYPE_BFLOAT16: return dtype::bfloat16; + case TGM_DTYPE_FLOAT32: return dtype::float32; + case TGM_DTYPE_FLOAT64: return dtype::float64; + case TGM_DTYPE_COMPLEX64: return dtype::complex64; + case TGM_DTYPE_COMPLEX128: return dtype::complex128; + case TGM_DTYPE_INT8: return dtype::int8; + case TGM_DTYPE_INT16: return dtype::int16; + case TGM_DTYPE_INT32: return dtype::int32; + case TGM_DTYPE_INT64: return dtype::int64; + case TGM_DTYPE_UINT8: return dtype::uint8; + case TGM_DTYPE_UINT16: return dtype::uint16; + case TGM_DTYPE_UINT32: return dtype::uint32; + case TGM_DTYPE_UINT64: return dtype::uint64; + case TGM_DTYPE_BITMASK: return dtype::bitmask; + } + // Defensive: an unknown discriminant from a future ABI decays to the + // zero variant, which is also what the C accessor reports on failure. + return dtype::float16; +} + +/// Map a C `tgm_byte_order` discriminant to the C++ byte_order enum. +[[nodiscard]] inline byte_order byte_order_from_c(tgm_byte_order o) noexcept { + switch (o) { + case TGM_BYTE_ORDER_LITTLE: return byte_order::little; + case TGM_BYTE_ORDER_BIG: return byte_order::big; + } + // Defensive: an unknown discriminant from a future ABI decays to the + // zero variant. + return byte_order::little; +} + +/// Map the C++ aggregate_hash_policy to the C discriminant. +[[nodiscard]] inline tgm_aggregate_hash_policy aggregate_hash_policy_to_c( + aggregate_hash_policy p) noexcept { + switch (p) { + case aggregate_hash_policy::automatic: return TGM_AGGREGATE_HASH_POLICY_AUTO; + case aggregate_hash_policy::none: return TGM_AGGREGATE_HASH_POLICY_NONE; + case aggregate_hash_policy::header: return TGM_AGGREGATE_HASH_POLICY_HEADER; + case aggregate_hash_policy::footer: return TGM_AGGREGATE_HASH_POLICY_FOOTER; + case aggregate_hash_policy::both: return TGM_AGGREGATE_HASH_POLICY_BOTH; + } + // Defensive: an unknown enumerator decays to the library default. + return TGM_AGGREGATE_HASH_POLICY_AUTO; +} + +/// Map the C++ compression_backend to the C discriminant. +[[nodiscard]] inline tgm_compression_backend compression_backend_to_c( + compression_backend b) noexcept { + switch (b) { + case compression_backend::automatic: return TGM_COMPRESSION_BACKEND_AUTO; + case compression_backend::ffi: return TGM_COMPRESSION_BACKEND_FFI; + case compression_backend::pure: return TGM_COMPRESSION_BACKEND_PURE; + } + // Defensive: an unknown enumerator decays to the library default. + return TGM_COMPRESSION_BACKEND_AUTO; +} + +/// Bounds-check an object index on behalf of the typed enum accessors. +/// +/// The C enum accessors have no spare code for failure: on a NULL handle or +/// an out-of-range index they record the reason in the thread-local error +/// slot and return their zero variant, which is indistinguishable from a +/// genuine `float16` / `little` answer. The paired **string** getter returns +/// NULL for exactly those inputs, so it is the unambiguous check — pass its +/// result as @p probe and the wrapper raises instead of handing back a +/// plausible-looking value. +inline void require_object_index(const char* probe, std::size_t index, const char* what) { + if (probe == nullptr) { + throw invalid_arg_error(TGM_ERROR_INVALID_ARG, + std::string(what) + ": object index " + + std::to_string(index) + " is out of range"); + } +} + +} // namespace detail + /// Options controlling scan_with_options() and scan_file_with_options(). /// /// Mirrors the C `TgmScanOptions` POD and `tensogram::ScanOptions`. A @@ -362,6 +534,20 @@ struct simple_packing_params { }; /// Options controlling message encoding. +/// +/// Maps onto the C `TgmEncodeOptions` POD (plus the `threads` argument the +/// C entry points take separately). Every field's default selects the +/// library default, so a default-constructed encode_options behaves exactly +/// like passing no options at all. +/// +/// The encode entry points — encode(), file::append() and +/// streaming_encoder — pick the narrowest C function that can carry the +/// options actually set: the plain one, the mask-aware one, or the full +/// `*_with_encode_options` one. No field is ever silently dropped, and a +/// caller who sets nothing new keeps the exact call path (and bytes) they +/// had before these knobs existed. encode_pre_encoded() is the exception +/// the C ABI imposes: it has no full-option entry point, so it rejects +/// `aggregate_hash` / `codec_backend` rather than ignoring them. struct encode_options { /// Hash algorithm name (e.g. "xxh3"). Empty string disables hashing. std::string hash_algo = "xxh3"; @@ -397,6 +583,20 @@ struct encode_options { /// 128. Set to 0 to disable the fallback. Negative values use the /// library default. std::ptrdiff_t small_mask_threshold_bytes = -1; + /// Where to write the aggregate hash frame. Ignored when hashing is + /// off (`hash_algo` empty or "none") — there is nothing to aggregate. + /// + /// aggregate_hash_policy::header and ::both are buffered-mode only; + /// streaming_encoder rejects them with tensogram::encoding_error. + aggregate_hash_policy aggregate_hash = aggregate_hash_policy::automatic; + /// Which codec implementation to prefer for szip / zstd. + /// + /// Maps to `TgmEncodeOptions::compression_backend` and to + /// `compression_backend=` / `compressionBackend` in the other bindings. + /// Spelled `codec_backend` here because a member named + /// `compression_backend` would shadow the enum type of the same name + /// inside this struct. + compression_backend codec_backend = compression_backend::automatic; }; /// Options controlling message decoding. @@ -467,6 +667,41 @@ inline mask_opts_holder build_mask_opts(const encode_options& opts) { h.value.small_mask_threshold_bytes = opts.small_mask_threshold_bytes; return h; } + +/// True when encode_options carries something only the full +/// `TgmEncodeOptions` entry points can express — the aggregate-hash +/// placement or the codec backend. +/// +/// The encode paths use this to escalate: with neither knob set they keep +/// calling the narrower C functions (`tgm_encode` / `tgm_encode_with_options` +/// and friends), so existing callers keep their exact behaviour; with either +/// set they switch to `*_with_encode_options`, so nothing is dropped. +[[nodiscard]] inline bool needs_encode_options(const encode_options& opts) noexcept { + return opts.aggregate_hash != aggregate_hash_policy::automatic + || opts.codec_backend != compression_backend::automatic; +} + +/// Build the full `TgmEncodeOptions` POD from encode_options. +/// +/// @warning The struct's `const char*` fields alias strings owned by @p opts, +/// so the returned value must not outlive it. +[[nodiscard]] inline TgmEncodeOptions build_encode_opts(const encode_options& opts) noexcept { + TgmEncodeOptions c_opts{}; + // The FFI convention is "name it to get it": a NULL hash means no hashing. + c_opts.hash = opts.hash_algo.empty() ? nullptr : opts.hash_algo.c_str(); + c_opts.aggregate_hash = aggregate_hash_policy_to_c(opts.aggregate_hash); + c_opts.compression_backend = compression_backend_to_c(opts.codec_backend); + c_opts.allow_nan = opts.allow_nan; + c_opts.allow_inf = opts.allow_inf; + c_opts.nan_mask_method = + opts.nan_mask_method.empty() ? nullptr : opts.nan_mask_method.c_str(); + c_opts.pos_inf_mask_method = + opts.pos_inf_mask_method.empty() ? nullptr : opts.pos_inf_mask_method.c_str(); + c_opts.neg_inf_mask_method = + opts.neg_inf_mask_method.empty() ? nullptr : opts.neg_inf_mask_method.c_str(); + c_opts.small_mask_threshold_bytes = opts.small_mask_threshold_bytes; + return c_opts; +} } // namespace detail // ============================================================ @@ -537,6 +772,20 @@ class decoded_object { return s ? s : ""; } + /// Dtype as a typed enumerator (e.g. dtype::float32) — the `switch`-able + /// companion to dtype_string(). + /// + /// Wraps `tgm_object_dtype_enum`. That accessor has no spare code to + /// signal failure, so this wrapper bounds-checks through the paired + /// string getter first and raises rather than returning the zero variant. + /// + /// @throws tensogram::invalid_arg_error If the object index is out of + /// range (dtype_string() returns "" for exactly the same input). + [[nodiscard]] tensogram::dtype dtype_enum() const { + detail::require_object_index(tgm_object_dtype(msg_, index_), index_, "dtype_enum"); + return detail::dtype_from_c(tgm_object_dtype_enum(msg_, index_)); + } + /// Object type string (e.g. "ndarray"). [[nodiscard]] std::string object_type() const { const char* s = tgm_object_type(msg_, index_); @@ -549,6 +798,20 @@ class decoded_object { return s ? s : ""; } + /// Byte order as a typed enumerator — the `switch`-able companion to + /// byte_order_string(). + /// + /// Wraps `tgm_object_byte_order_enum`; see dtype_enum() for why the + /// bounds check goes through the string getter. + /// + /// @throws tensogram::invalid_arg_error If the object index is out of + /// range (byte_order_string() returns "" for the same input). + [[nodiscard]] tensogram::byte_order byte_order_enum() const { + detail::require_object_index(tgm_object_byte_order(msg_, index_), index_, + "byte_order_enum"); + return detail::byte_order_from_c(tgm_object_byte_order_enum(msg_, index_)); + } + /// Encoding pipeline string (e.g. "none", "simple_packing"). [[nodiscard]] std::string encoding() const { const char* s = tgm_payload_encoding(msg_, index_); @@ -1288,6 +1551,60 @@ inline metadata message::get_metadata() const { return metadata(raw); } +// ============================================================ +// Remote sources — is_remote_url / remote_scan_options +// ============================================================ + +/// Reader-side scan-walker options for file::open_remote(). +/// +/// Mirrors the C `TgmRemoteScanOptions` POD and +/// `tensogram::RemoteScanOptions`. A default-constructed +/// remote_scan_options requests the library defaults. +struct remote_scan_options { + /// Enable the meet-in-the-middle (bidirectional) remote walk, which + /// pairs forward preamble fetches with backward postamble fetches and + /// roughly halves wall-clock layout discovery on real networks. + /// `false` forces a forward-only walk. + bool bidirectional = true; +}; + +namespace detail { + +/// Reject a string that cannot survive the trip through a C string. +/// +/// The C ABI takes NUL-terminated `const char*`, so an interior NUL would +/// silently truncate the value — opening a *different* object than the +/// caller named. std::string / std::string_view can carry one, so the +/// wrapper checks rather than letting the truncation happen. +inline void require_nul_free(std::string_view value, const char* what) { + if (value.find('\0') != std::string_view::npos) { + throw invalid_arg_error(TGM_ERROR_INVALID_ARG, + std::string(what) + " must not contain an embedded NUL"); + } +} + +} // namespace detail + +/// True when @p source is a URL this build can open remotely. +/// +/// Wraps `tgm_is_remote_url`, binding `tensogram::is_remote_url`. The +/// recognised schemes are `s3`, `s3a`, `gs`, `az`, `azure`, `http` and +/// `https`, compared case-insensitively. Plain paths and `file://` URLs are +/// **not** remote — they belong to the local backend (file::open()). +/// +/// Returns false for **every** input when the C API was built without the +/// `remote` Cargo feature: such a build genuinely cannot open any remote +/// URL, so "not remote for me" is the honest answer. +/// +/// @return true if @p source names a remote object this build can open. +/// Never throws — a source with an embedded NUL is simply not a +/// remote URL. +[[nodiscard]] inline bool is_remote_url(std::string_view source) { + if (source.find('\0') != std::string_view::npos) return false; + const std::string src(source); + return tgm_is_remote_url(src.c_str()); +} + // ============================================================ // file — RAII wrapper for tgm_file_t // ============================================================ @@ -1316,6 +1633,61 @@ class file { return file(raw); } + /// Open a remote `.tgm` (S3 / GCS / Azure / HTTP) for **synchronous** + /// reading. + /// + /// Wraps `tgm_file_open_remote`. The result is an ordinary file, so the + /// whole existing API — message_count(), read_message(), + /// decode_message(), file_iterator, … — works unchanged against the + /// remote source, and the handle closes the same way. + /// + /// @param source Remote URL (see is_remote_url() for the + /// recognised schemes). + /// @param storage_options Backend options (credentials, region, + /// endpoint, …) forwarded verbatim to the + /// object-store backend. + /// @param opts Scan-walker options; the default requests the + /// bidirectional walk. + /// @throws tensogram::invalid_arg_error If @p source or any storage + /// option contains an embedded NUL (it could not be passed to + /// the C ABI without silently truncating). + /// @throws tensogram::remote_error If the URL is unparseable, the object + /// is missing, a storage option is rejected, the transport + /// fails — or the C API was built without the `remote` Cargo + /// feature, in which case the message says how to enable it. + [[nodiscard]] static file open_remote( + std::string_view source, + const std::map& storage_options = {}, + remote_scan_options opts = {}) + { + detail::require_nul_free(source, "remote source"); + const std::string source_str(source); + + // The C call takes parallel key / value arrays of `const char*`. + // The map owns the strings, so the pointers stay valid for the call. + std::vector keys; + std::vector values; + keys.reserve(storage_options.size()); + values.reserve(storage_options.size()); + for (const auto& [key, value] : storage_options) { + detail::require_nul_free(key, "storage option key"); + detail::require_nul_free(value, "storage option value"); + keys.push_back(key.c_str()); + values.push_back(value.c_str()); + } + + TgmRemoteScanOptions c_opts{}; + c_opts.bidirectional = opts.bidirectional; + + tgm_file_t* raw = nullptr; + detail::check(tgm_file_open_remote( + source_str.c_str(), + keys.empty() ? nullptr : keys.data(), + values.empty() ? nullptr : values.data(), + keys.size(), &c_opts, &raw)); + return file(raw); + } + file(file&&) noexcept = default; file& operator=(file&&) noexcept = default; ~file() = default; @@ -1373,7 +1745,13 @@ class file { const char* hash = opts.hash_algo.empty() ? nullptr : opts.hash_algo.c_str(); auto mask_holder = detail::build_mask_opts(opts); - if (mask_holder.active) { + if (detail::needs_encode_options(opts)) { + const TgmEncodeOptions c_opts = detail::build_encode_opts(opts); + detail::check(tgm_file_append_with_encode_options( + handle_.get(), metadata_json.c_str(), + sg.ptrs.data(), sg.lens.data(), + objects.size(), opts.threads, &c_opts)); + } else if (mask_holder.active) { detail::check(tgm_file_append_with_options( handle_.get(), metadata_json.c_str(), sg.ptrs.data(), sg.lens.data(), @@ -1553,6 +1931,382 @@ class object_iterator { std::unique_ptr handle_; }; +// ============================================================ +// Frame walker — frame_type / frame / frame_range +// ============================================================ + +/// The kind of a wire frame, mirroring the C `tgm_frame_type` enum. +/// +/// The enumerator values **are** the wire's frame-type numbers (see +/// `plans/WIRE_FORMAT.md` §4), not a binding invention. Type 4 is +/// reserved — it held the obsolete v2 data-object layout — and therefore +/// has no enumerator, which is why the sequence skips from 3 to 5. +enum class frame_type : std::uint16_t { + /// CBOR global metadata, written in the header (random-access mode). + header_metadata = TGM_FRAME_TYPE_HEADER_METADATA, + /// Object index, written in the header (random-access mode). + header_index = TGM_FRAME_TYPE_HEADER_INDEX, + /// Aggregate hash frame, written in the header. + header_hash = TGM_FRAME_TYPE_HEADER_HASH, + /// Aggregate hash frame, written in the footer. + footer_hash = TGM_FRAME_TYPE_FOOTER_HASH, + /// Object index, written in the footer (streaming mode). + footer_index = TGM_FRAME_TYPE_FOOTER_INDEX, + /// CBOR global metadata, written in the footer (streaming mode). + footer_metadata = TGM_FRAME_TYPE_FOOTER_METADATA, + /// Per-object metadata frame immediately preceding a data-object frame. + preceder_metadata = TGM_FRAME_TYPE_PRECEDER_METADATA, + /// N-dimensional tensor data-object frame — the only data-object type + /// in v3, and the only frame type with a 20-byte footer. + ntensor = TGM_FRAME_TYPE_NTENSOR, +}; + +namespace detail { + +/// Map the C `tgm_frame_type` discriminant to the C++ frame_type enum. +/// +/// The two enums carry identical values (both are the wire number), so this +/// is an exact, total conversion rather than a lookup table — an unknown +/// discriminant from a future ABI is preserved as its number instead of +/// being silently folded into a known variant. +[[nodiscard]] inline frame_type frame_type_from_c(tgm_frame_type t) noexcept { + return static_cast(static_cast(t)); +} + +} // namespace detail + +/// One frame's structural description plus a **borrowed** view of its +/// content — the C++ face of the C `TgmFrame` POD. +/// +/// A frame is a small value type (copyable, cheap) yielded by frame_range. +/// +/// @warning Lifetime: payload() / payload_data() point **into the message +/// buffer** passed to frames() — a view, never a copy, with nothing +/// to free. They stay valid for exactly as long as that buffer +/// does, independently of the range that produced them: destroying +/// the frame_range does **not** invalidate them. Copy the bytes +/// out if you need them to outlive the message buffer. +class frame { +public: + /// Which kind of frame this is. + [[nodiscard]] frame_type type() const noexcept { + return detail::frame_type_from_c(raw_.frame_type); + } + + /// Frame-type-specific version field from the frame header. + [[nodiscard]] std::uint16_t version() const noexcept { return raw_.version; } + + /// Raw 16-bit frame flags; bit 1 is `HASH_PRESENT` (see has_hash()). + [[nodiscard]] std::uint16_t flags() const noexcept { return raw_.flags; } + + /// Byte offset of the frame header, relative to the start of the message. + [[nodiscard]] std::size_t offset() const noexcept { return raw_.offset; } + + /// Whole-frame span in bytes: frame header through `ENDF`, excluding any + /// alignment padding that follows. + [[nodiscard]] std::size_t length() const noexcept { return raw_.length; } + + /// Borrowed pointer to the content bytes — everything between the + /// 16-byte frame header and the type-specific footer (20 bytes for + /// frame_type::ntensor, 12 for every other type). + /// @warning Borrows the caller's message buffer; see the class warning. + [[nodiscard]] const std::uint8_t* payload_data() const noexcept { return raw_.payload; } + + /// Length of the borrowed content bytes. + [[nodiscard]] std::size_t payload_size() const noexcept { return raw_.payload_len; } + + /// The content bytes as a std::string_view (a byte view — the payload is + /// binary CBOR or tensor data, not text). + /// + /// The project targets C++17, so this is the borrowing byte-range type + /// available; it is built from `ptr + len` and never `strlen`, so interior + /// NUL bytes are preserved. + /// @warning Borrows the caller's message buffer; see the class warning. + [[nodiscard]] std::string_view payload() const noexcept { + if (raw_.payload == nullptr) return {}; + return std::string_view(reinterpret_cast(raw_.payload), + raw_.payload_len); + } + + /// True if this frame's `HASH_PRESENT` flag is set, i.e. its inline hash + /// slot holds a meaningful digest (`plans/WIRE_FORMAT.md` §2.5). + /// + /// This is the authoritative answer for a single frame — + /// message_header::has_hashes_present() is only an advisory + /// message-wide summary. + [[nodiscard]] bool has_hash() const noexcept { return tgm_frame_has_hash(&raw_); } + + /// True if this frame carries a data object (frame_type::ntensor is the + /// only data-object type in v3). + [[nodiscard]] bool is_data_object() const noexcept { + return type() == frame_type::ntensor; + } + +private: + friend class frame_range; + + frame() = default; + explicit frame(const TgmFrame& raw) noexcept : raw_(raw) {} + TgmFrame raw_{}; +}; + +/// A lazy, single-pass range over the frames of one message. +/// +/// Returned by frames(). Owns the underlying C cursor (RAII, move-only) and +/// pulls exactly one frame per `++` — nothing is materialised up front, so a +/// caller that only needs the first frames never pays for the rest. +/// +/// Supports range-for: +/// @code +/// for (const auto& f : tensogram::frames(msg.data(), msg.size())) { +/// if (f.is_data_object()) { ... } +/// } +/// @endcode +/// +/// @warning **Lifetime**: the range borrows the message buffer — the buffer +/// must outlive the range and must not be moved, reallocated, or +/// mutated while the range lives. Frames yielded by the walk keep +/// pointing into that buffer and stay valid after the range is +/// destroyed (see the frame class documentation). +/// +/// @note **End vs malformed**: the underlying cursor stops for two different +/// reasons. A clean end simply terminates the loop; a truncated or +/// inconsistent frame chain throws tensogram::framing_error from the +/// increment that discovers it, after the intact frames have been +/// yielded. A buffer whose preamble does not parse at all throws from +/// frames() itself. +class frame_range { +public: + /// End-of-walk marker. C++17 permits a range-for `end()` of a different + /// type from `begin()`, which lets the cursor answer "am I done?" from + /// its own state instead of synthesising a comparable end iterator. + struct sentinel {}; + + /// Input iterator over the frames of a message. + /// + /// Single-pass by construction: every copy shares the one C cursor owned + /// by the frame_range, exactly like std::istream_iterator. + class iterator { + public: + using value_type = frame; + using difference_type = std::ptrdiff_t; + using pointer = const frame*; + using reference = const frame&; + using iterator_category = std::input_iterator_tag; + + /// Construct an exhausted iterator (compares equal to any sentinel). + iterator() = default; + + [[nodiscard]] const frame& operator*() const noexcept { return current_; } + [[nodiscard]] const frame* operator->() const noexcept { return ¤t_; } + + /// Pull the next frame. + /// @throws tensogram::framing_error If the frame chain is malformed. + iterator& operator++() { advance(); return *this; } + + /// Post-increment returns void, as C++20's `std::input_iterator` + /// permits: every copy shares the one C cursor, so a returned copy + /// could not be advanced independently of this one. Use the + /// pre-increment form. + /// @throws tensogram::framing_error If the frame chain is malformed. + void operator++(int) { advance(); } + + [[nodiscard]] bool operator==(sentinel) const noexcept { return done_; } + [[nodiscard]] bool operator!=(sentinel) const noexcept { return !done_; } + [[nodiscard]] friend bool operator==(sentinel s, const iterator& it) noexcept { + return it == s; + } + [[nodiscard]] friend bool operator!=(sentinel s, const iterator& it) noexcept { + return it != s; + } + /// Two iterators are equal when they share a cursor and its state. + [[nodiscard]] bool operator==(const iterator& other) const noexcept { + return it_ == other.it_ && done_ == other.done_; + } + [[nodiscard]] bool operator!=(const iterator& other) const noexcept { + return !(*this == other); + } + + private: + friend class frame_range; + + explicit iterator(tgm_frame_iter_t* it) : it_(it), done_(false) { advance(); } + + /// Pull one frame, translating the C cursor's single `false` return + /// into the two outcomes it actually stands for. + /// + /// `tgm_frame_iter_next` clears the thread-local last error on a + /// clean exhaustion and sets it on a malformed frame, so the error + /// slot — not the return value — is what tells the two apart. + void advance() { + if (done_ || it_ == nullptr) { + done_ = true; + return; + } + TgmFrame raw{}; + if (tgm_frame_iter_next(it_, &raw)) { + current_ = frame(raw); + return; + } + done_ = true; + current_ = frame(); + if (const char* msg = tgm_last_error()) { + throw framing_error(TGM_ERROR_FRAMING, msg); + } + } + + /// Borrowed from the owning frame_range — the cursor's address is + /// stable across a move of the range, so iterators survive it. + tgm_frame_iter_t* it_ = nullptr; + frame current_{}; + bool done_ = true; + }; + + /// Start a walk over the frames of the message in `[msg, msg + len)`. + /// + /// @throws tensogram::invalid_arg_error If @p msg is null. + /// @throws tensogram::framing_error If the preamble does not parse + /// (truncated buffer, wrong magic, unsupported version). + frame_range(const std::uint8_t* msg, std::size_t len) { + if (msg == nullptr) { + throw invalid_arg_error(TGM_ERROR_INVALID_ARG, + "frames: message buffer is null"); + } + tgm_frame_iter_t* raw = tgm_frame_iter_create(msg, len); + if (raw == nullptr) { + const char* err = tgm_last_error(); + throw framing_error(TGM_ERROR_FRAMING, + err ? err : "message preamble does not parse"); + } + handle_.reset(raw); + } + + frame_range(frame_range&&) noexcept = default; + frame_range& operator=(frame_range&&) noexcept = default; + ~frame_range() = default; + + frame_range(const frame_range&) = delete; + frame_range& operator=(const frame_range&) = delete; + + /// Begin the walk, pulling the first frame. + /// + /// Single-pass: calling begin() a second time resumes where the previous + /// iterator stopped rather than restarting the walk. + /// @throws tensogram::framing_error If the first frame is malformed. + [[nodiscard]] iterator begin() { return iterator(handle_.get()); } + + /// The end-of-walk sentinel. + [[nodiscard]] sentinel end() const noexcept { return sentinel{}; } + +private: + struct deleter { + void operator()(tgm_frame_iter_t* p) const noexcept { tgm_frame_iter_free(p); } + }; + std::unique_ptr handle_; +}; + +/// Walk the frames of a single message, lazily. +/// +/// Wraps `tgm_frame_iter_create`, binding `tensogram::frames`. @p msg must +/// point at the start of a message (the `TENSOGRM` preamble magic) — +/// typically a slice located with scan(). Only the type 1–9 `FR` frames are +/// yielded; the preamble and postamble are not frames, use +/// read_message_header() for the envelope. +/// +/// @param msg Wire-format bytes of a single message. +/// @param len Length of @p msg. +/// @return A lazy range; see frame_range for the lifetime and +/// end-vs-malformed contracts. +/// @throws tensogram::invalid_arg_error If @p msg is null. +/// @throws tensogram::framing_error If the preamble does not parse. +[[nodiscard]] inline frame_range frames(const std::uint8_t* msg, std::size_t len) { + return frame_range(msg, len); +} + +// ============================================================ +// message_header — the message envelope, without walking frames +// ============================================================ + +class message_header; +[[nodiscard]] inline message_header read_message_header(const std::uint8_t* msg, + std::size_t len); + +/// A message's envelope (the 24-byte preamble) as a value type. +/// +/// Produced by read_message_header(). The eight `has_*` accessors are the +/// preamble's structural flags decoded into named predicates, so callers +/// never touch a raw bitset. Together they say whether a message is +/// *random-access* (metadata / index / hashes in the **header**) or +/// *streaming* (in the **footer**) without reading a single frame. +class message_header { +public: + /// Wire-format version (tensogram::wire_version for messages this build + /// writes). + [[nodiscard]] std::uint16_t version() const noexcept { return raw_.version; } + + /// Total message length in bytes, preamble through postamble, or `0` if + /// a streaming writer never back-filled it. Zero is not an error — it + /// means "unknown at write time". + [[nodiscard]] std::uint64_t total_length() const noexcept { return raw_.total_length; } + + /// A frame_type::header_metadata frame is present (random-access mode). + [[nodiscard]] bool has_header_metadata() const noexcept { + return raw_.has_header_metadata; + } + /// A frame_type::footer_metadata frame is present (streaming mode). + [[nodiscard]] bool has_footer_metadata() const noexcept { + return raw_.has_footer_metadata; + } + /// A frame_type::header_index frame is present. + [[nodiscard]] bool has_header_index() const noexcept { return raw_.has_header_index; } + /// A frame_type::footer_index frame is present. + [[nodiscard]] bool has_footer_index() const noexcept { return raw_.has_footer_index; } + /// A frame_type::header_hash frame is present. + [[nodiscard]] bool has_header_hashes() const noexcept { return raw_.has_header_hashes; } + /// A frame_type::footer_hash frame is present. + [[nodiscard]] bool has_footer_hashes() const noexcept { return raw_.has_footer_hashes; } + /// At least one frame_type::preceder_metadata frame appears in the body. + /// + /// **Advisory in streaming mode**: the encoder writes the preamble before + /// it knows whether any preceder will follow, so `true` does not + /// guarantee a frame. Only "frame present ⇒ flag set" holds there; in + /// buffered mode the flag is exact. + [[nodiscard]] bool has_preceder_metadata() const noexcept { + return raw_.has_preceder_metadata; + } + /// Advisory: every frame in this message has its per-frame + /// `HASH_PRESENT` bit set. For any single frame, frame::has_hash() + /// stays authoritative. + [[nodiscard]] bool has_hashes_present() const noexcept { + return raw_.has_hashes_present; + } + +private: + friend message_header tensogram::read_message_header(const std::uint8_t*, std::size_t); + + message_header() = default; + TgmMessageHeader raw_{}; +}; + +/// Read a message's envelope without walking its frames. +/// +/// Wraps `tgm_message_header`, binding `tensogram::message_header`. @p msg +/// must point at the start of a message (the `TENSOGRM` preamble magic) — +/// typically a slice located with scan(). +/// +/// @param msg Wire-format bytes of a single message. +/// @param len Length of @p msg. +/// @return The decoded envelope. +/// @throws tensogram::invalid_arg_error If @p msg is null. +/// @throws tensogram::framing_error If the preamble does not parse +/// (truncated message, wrong magic, unsupported version). +[[nodiscard]] inline message_header read_message_header(const std::uint8_t* msg, + std::size_t len) { + message_header header; + detail::check(tgm_message_header(msg, len, &header.raw_)); + return header; +} + // ============================================================ // streaming_encoder — RAII wrapper for tgm_streaming_encoder_t // ============================================================ @@ -1570,6 +2324,12 @@ class streaming_encoder { /// @param path Output file path (created/truncated). /// @param metadata_json JSON with "version" and optional extra keys. /// @param opts Encoding options (hash algorithm, etc.). + /// @throws tensogram::encoding_error If `opts.aggregate_hash` asks for a + /// header-side placement (aggregate_hash_policy::header or + /// ::both): a streaming writer emits its header before any data + /// object exists, so the per-object hashes are not yet known. + /// Use aggregate_hash_policy::automatic (which resolves to the + /// footer when streaming) or ::footer. streaming_encoder(const std::string& path, const std::string& metadata_json, const encode_options& opts = {}) { @@ -1577,7 +2337,11 @@ class streaming_encoder { const char* hash = opts.hash_algo.empty() ? nullptr : opts.hash_algo.c_str(); auto mask_holder = detail::build_mask_opts(opts); - if (mask_holder.active) { + if (detail::needs_encode_options(opts)) { + const TgmEncodeOptions c_opts = detail::build_encode_opts(opts); + detail::check(tgm_streaming_encoder_create_with_encode_options( + path.c_str(), metadata_json.c_str(), opts.threads, &c_opts, &raw)); + } else if (mask_holder.active) { detail::check(tgm_streaming_encoder_create_with_options( path.c_str(), metadata_json.c_str(), hash, opts.threads, &mask_holder.value, &raw)); @@ -1682,7 +2446,16 @@ class streaming_encoder { tgm_bytes_t bytes{}; auto mask_holder = detail::build_mask_opts(opts); const TgmEncodeMaskOptions* mask_ptr = mask_holder.active ? &mask_holder.value : nullptr; - if (mask_ptr != nullptr) { + if (detail::needs_encode_options(opts)) { + // Aggregate-hash placement / codec backend live only in the full + // option struct, which also carries the hash algorithm and the mask + // policy — so there is no separate `hash_algo` argument here. + const TgmEncodeOptions c_opts = detail::build_encode_opts(opts); + detail::check(tgm_encode_with_encode_options(metadata_json.c_str(), + sg.ptrs.data(), sg.lens.data(), + objects.size(), opts.threads, + &c_opts, &bytes)); + } else if (mask_ptr != nullptr) { detail::check(tgm_encode_with_options(metadata_json.c_str(), sg.ptrs.data(), sg.lens.data(), objects.size(), hash, opts.threads, @@ -1728,18 +2501,38 @@ class streaming_encoder { /// @param objects Vector of (pointer, length) pairs pointing at /// already-encoded payload bytes — one per descriptor /// entry. -/// @param opts Encoding options (hash algorithm, etc.). +/// @param opts Encoding options (hash algorithm, threads). The +/// mask fields do not apply to opaque pre-encoded +/// bytes and are ignored. /// @return The encoded message as a byte vector. +/// @throws tensogram::invalid_arg_error If `opts.aggregate_hash` or +/// `opts.codec_backend` is set to a non-default value: the C ABI +/// has no pre-encoded entry point that carries them, so honouring +/// them here is impossible and ignoring them would emit a message +/// that contradicts the request. Use encode() for those. [[nodiscard]] inline std::vector encode_pre_encoded( const std::string& metadata_json, const std::vector>& objects, const encode_options& opts = {}) { - // Strict-finite flags are raw-input-only — pre-encoded bytes are - // opaque to the library and cannot be meaningfully scanned for - // NaN/Inf. The underlying C FFI does not accept these flags, but - // we catch the case here to give the C++ caller a clear error - // rather than silently discarding their intent. + // The mask fields are raw-input-only — pre-encoded bytes are opaque to + // the library and cannot be meaningfully scanned for NaN / Inf — and + // `tgm_encode_pre_encoded` accordingly takes no mask options, so they + // are inapplicable here. + // + // The aggregate-hash placement and the codec backend are different: + // they describe the *message* the encoder writes, not the payload + // bytes, yet the C ABI has no pre-encoded entry point that carries + // them. Accepting them would emit a message that contradicts what the + // caller asked for, so the mismatch is reported instead of dropped. + if (detail::needs_encode_options(opts)) { + throw invalid_arg_error( + TGM_ERROR_INVALID_ARG, + "encode_pre_encoded: encode_options::aggregate_hash and " + "::codec_backend are not supported on the pre-encoded path " + "(the C ABI has no pre-encoded entry point that carries them); " + "use encode() for those, or leave both at their default"); + } detail::scatter_gather sg(objects); const char* hash = opts.hash_algo.empty() ? nullptr : opts.hash_algo.c_str(); tgm_bytes_t bytes{}; diff --git a/cpp/tests/CMakeLists.txt b/cpp/tests/CMakeLists.txt index cd1e43a8..802e4aee 100644 --- a/cpp/tests/CMakeLists.txt +++ b/cpp/tests/CMakeLists.txt @@ -28,6 +28,10 @@ set(TENSOGRAM_TEST_SOURCES test_canonical_cbor.cpp test_scan_family.cpp test_doctor.cpp + test_frame_walk.cpp + test_typed_enums.cpp + test_encode_options.cpp + test_remote.cpp ) if(TENSOGRAM_ASYNC) diff --git a/cpp/tests/test_encode_options.cpp b/cpp/tests/test_encode_options.cpp new file mode 100644 index 00000000..f1bc15f7 --- /dev/null +++ b/cpp/tests/test_encode_options.cpp @@ -0,0 +1,385 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +// SPDX-License-Identifier: Apache-2.0 +// Copyright 2026 ECMWF +// +// Tests for the full encode-side option set (TgmEncodeOptions): +// * tensogram::aggregate_hash_policy — where the aggregate hash frame goes +// * tensogram::compression_backend — which codec implementation to prefer +// wired through encode(), file::append() and streaming_encoder. +// +// The aggregate-hash assertions close the loop with this binding's own frame +// walker: the placement knob is only meaningful if the frames actually land +// where it says, so every case is verified by walking the encoded message. + +#include +#include +#include "test_helpers.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include + +using test_helpers::TempFile; + +namespace { + +const char* const kDescriptor = + R"({"type":"ndarray","ndim":1,"shape":[4],"strides":[4],"dtype":"float32",)" + R"("byte_order":"little","encoding":"none","filter":"none","compression":"none"})"; + +const std::vector& sample_values() { + static const std::vector values{1.0f, 2.0f, 3.0f, 4.0f}; + return values; +} + +std::vector> sample_objects() { + return {{reinterpret_cast(sample_values().data()), + sample_values().size() * sizeof(float)}}; +} + +std::string one_object_json() { + return std::string(R"({"descriptors":[)") + kDescriptor + "]}"; +} + +/// Collect the frame types of an encoded message. +std::vector frame_types(const std::vector& msg) { + std::vector types; + for (const auto& f : tensogram::frames(msg.data(), msg.size())) { + types.push_back(f.type()); + } + return types; +} + +bool contains(const std::vector& types, tensogram::frame_type t) { + return std::find(types.begin(), types.end(), t) != types.end(); +} + +std::size_t count_of(const std::vector& types, tensogram::frame_type t) { + return static_cast(std::count(types.begin(), types.end(), t)); +} + +} // namespace + +// --------------------------------------------------------------------------- +// The policy enums mirror the C ABI +// --------------------------------------------------------------------------- + +TEST(EncodeOptionsTest, PolicyEnumsMirrorTheCValues) { + using tensogram::aggregate_hash_policy; + EXPECT_EQ(static_cast(aggregate_hash_policy::automatic), + static_cast(TGM_AGGREGATE_HASH_POLICY_AUTO)); + EXPECT_EQ(static_cast(aggregate_hash_policy::none), + static_cast(TGM_AGGREGATE_HASH_POLICY_NONE)); + EXPECT_EQ(static_cast(aggregate_hash_policy::header), + static_cast(TGM_AGGREGATE_HASH_POLICY_HEADER)); + EXPECT_EQ(static_cast(aggregate_hash_policy::footer), + static_cast(TGM_AGGREGATE_HASH_POLICY_FOOTER)); + EXPECT_EQ(static_cast(aggregate_hash_policy::both), + static_cast(TGM_AGGREGATE_HASH_POLICY_BOTH)); + + using tensogram::compression_backend; + EXPECT_EQ(static_cast(compression_backend::automatic), + static_cast(TGM_COMPRESSION_BACKEND_AUTO)); + EXPECT_EQ(static_cast(compression_backend::ffi), + static_cast(TGM_COMPRESSION_BACKEND_FFI)); + EXPECT_EQ(static_cast(compression_backend::pure), + static_cast(TGM_COMPRESSION_BACKEND_PURE)); +} + +TEST(EncodeOptionsTest, DefaultsSelectTheLibraryBehaviour) { + const tensogram::encode_options opts; + EXPECT_EQ(opts.aggregate_hash, tensogram::aggregate_hash_policy::automatic); + EXPECT_EQ(opts.codec_backend, tensogram::compression_backend::automatic); +} + +// --------------------------------------------------------------------------- +// encode() — aggregate hash placement, verified by walking the frames +// --------------------------------------------------------------------------- + +TEST(EncodeOptionsTest, DefaultPlacementPutsTheAggregateHashInTheHeader) { + // AUTO resolves to the header when buffering, matching what plain + // encode() has always produced. + auto types = frame_types(tensogram::encode(one_object_json(), sample_objects())); + EXPECT_TRUE(contains(types, tensogram::frame_type::header_hash)); + EXPECT_FALSE(contains(types, tensogram::frame_type::footer_hash)); +} + +TEST(EncodeOptionsTest, AggregateHashBothWritesHeaderAndFooterHashFrames) { + tensogram::encode_options opts; + opts.aggregate_hash = tensogram::aggregate_hash_policy::both; + auto msg = tensogram::encode(one_object_json(), sample_objects(), opts); + + auto types = frame_types(msg); + EXPECT_EQ(count_of(types, tensogram::frame_type::header_hash), 1u); + EXPECT_EQ(count_of(types, tensogram::frame_type::footer_hash), 1u); + + const auto header = tensogram::read_message_header(msg.data(), msg.size()); + EXPECT_TRUE(header.has_header_hashes()); + EXPECT_TRUE(header.has_footer_hashes()); +} + +TEST(EncodeOptionsTest, AggregateHashFooterMovesTheFrameOutOfTheHeader) { + tensogram::encode_options opts; + opts.aggregate_hash = tensogram::aggregate_hash_policy::footer; + auto msg = tensogram::encode(one_object_json(), sample_objects(), opts); + + auto types = frame_types(msg); + EXPECT_FALSE(contains(types, tensogram::frame_type::header_hash)); + EXPECT_EQ(count_of(types, tensogram::frame_type::footer_hash), 1u); + // The footer hash frame really is after the data object. + const auto data_pos = std::find(types.begin(), types.end(), tensogram::frame_type::ntensor); + const auto hash_pos = std::find(types.begin(), types.end(), + tensogram::frame_type::footer_hash); + EXPECT_LT(data_pos - types.begin(), hash_pos - types.begin()); +} + +TEST(EncodeOptionsTest, AggregateHashHeaderIsExplicitlySelectable) { + tensogram::encode_options opts; + opts.aggregate_hash = tensogram::aggregate_hash_policy::header; + auto types = frame_types(tensogram::encode(one_object_json(), sample_objects(), opts)); + EXPECT_EQ(count_of(types, tensogram::frame_type::header_hash), 1u); + EXPECT_FALSE(contains(types, tensogram::frame_type::footer_hash)); +} + +TEST(EncodeOptionsTest, AggregateHashNoneKeepsThePerFrameInlineHashes) { + tensogram::encode_options opts; + opts.aggregate_hash = tensogram::aggregate_hash_policy::none; + auto msg = tensogram::encode(one_object_json(), sample_objects(), opts); + + auto types = frame_types(msg); + EXPECT_FALSE(contains(types, tensogram::frame_type::header_hash)); + EXPECT_FALSE(contains(types, tensogram::frame_type::footer_hash)); + // "No aggregate frame" is not "no hashing": the per-frame slots stay. + for (const auto& f : tensogram::frames(msg.data(), msg.size())) { + EXPECT_TRUE(f.has_hash()); + } + EXPECT_TRUE(tensogram::read_message_header(msg.data(), msg.size()).has_hashes_present()); +} + +TEST(EncodeOptionsTest, PlacementIsMootWhenHashingIsOff) { + tensogram::encode_options opts; + opts.hash_algo = ""; + opts.aggregate_hash = tensogram::aggregate_hash_policy::both; + auto msg = tensogram::encode(one_object_json(), sample_objects(), opts); + + auto types = frame_types(msg); + EXPECT_FALSE(contains(types, tensogram::frame_type::header_hash)); + EXPECT_FALSE(contains(types, tensogram::frame_type::footer_hash)); + EXPECT_FALSE(tensogram::read_message_header(msg.data(), msg.size()).has_hashes_present()); +} + +TEST(EncodeOptionsTest, MessagesStillDecodeUnderEveryPlacement) { + using tensogram::aggregate_hash_policy; + for (auto policy : {aggregate_hash_policy::automatic, aggregate_hash_policy::none, + aggregate_hash_policy::header, aggregate_hash_policy::footer, + aggregate_hash_policy::both}) { + tensogram::encode_options opts; + opts.aggregate_hash = policy; + auto msg = tensogram::encode(one_object_json(), sample_objects(), opts); + auto decoded = tensogram::decode(msg.data(), msg.size()); + ASSERT_EQ(decoded.num_objects(), 1u); + const auto obj = decoded.object(0); + ASSERT_EQ(obj.element_count(), sample_values().size()); + EXPECT_EQ(obj.data_as()[3], 4.0f); + } +} + +// --------------------------------------------------------------------------- +// compression_backend +// --------------------------------------------------------------------------- + +TEST(EncodeOptionsTest, EveryCodecBackendRoundTripsLosslessly) { + // The compressed bytes may differ between implementations; the decoded + // payload must not. + const std::string json = + R"({"descriptors":[{"type":"ndarray","ndim":1,"shape":[64],"strides":[4],)" + R"("dtype":"float32","byte_order":"little","encoding":"none","filter":"none",)" + R"("compression":"zstd"}]})"; + std::vector values(64); + for (std::size_t i = 0; i < values.size(); ++i) values[i] = static_cast(i); + std::vector> objects{ + {reinterpret_cast(values.data()), values.size() * sizeof(float)}}; + + using tensogram::compression_backend; + for (auto backend : {compression_backend::automatic, compression_backend::ffi, + compression_backend::pure}) { + tensogram::encode_options opts; + opts.codec_backend = backend; + auto msg = tensogram::encode(json, objects, opts); + auto decoded = tensogram::decode(msg.data(), msg.size()); + ASSERT_EQ(decoded.num_objects(), 1u); + const auto obj = decoded.object(0); + ASSERT_EQ(obj.element_count(), values.size()); + EXPECT_EQ(0, std::memcmp(obj.data(), values.data(), values.size() * sizeof(float))) + << "backend " << static_cast(backend); + EXPECT_EQ(obj.compression(), "zstd"); + } +} + +// --------------------------------------------------------------------------- +// file::append +// --------------------------------------------------------------------------- + +TEST(EncodeOptionsTest, FileAppendHonoursTheAggregateHashPolicy) { + TempFile tmp; + tensogram::encode_options opts; + opts.aggregate_hash = tensogram::aggregate_hash_policy::both; + { + auto f = tensogram::file::create(tmp.path); + f.append(one_object_json(), sample_objects(), opts); + } + auto f = tensogram::file::open(tmp.path); + ASSERT_EQ(f.message_count(), 1u); + auto raw = f.read_message(0); + auto types = frame_types(raw); + EXPECT_EQ(count_of(types, tensogram::frame_type::header_hash), 1u); + EXPECT_EQ(count_of(types, tensogram::frame_type::footer_hash), 1u); +} + +TEST(EncodeOptionsTest, FileAppendHonoursTheCodecBackend) { + TempFile tmp; + tensogram::encode_options opts; + opts.codec_backend = tensogram::compression_backend::pure; + { + auto f = tensogram::file::create(tmp.path); + f.append(one_object_json(), sample_objects(), opts); + } + auto f = tensogram::file::open(tmp.path); + auto decoded = f.decode_message(0); + ASSERT_EQ(decoded.num_objects(), 1u); + EXPECT_EQ(decoded.object(0).data_as()[0], 1.0f); +} + +// --------------------------------------------------------------------------- +// streaming_encoder +// --------------------------------------------------------------------------- + +TEST(EncodeOptionsTest, StreamingRejectsHeaderSideAggregateHashPlacement) { + // A streaming writer emits its header before any data object exists, so + // the per-object hashes are not yet known. + using tensogram::aggregate_hash_policy; + for (auto policy : {aggregate_hash_policy::header, aggregate_hash_policy::both}) { + TempFile tmp; + tensogram::encode_options opts; + opts.aggregate_hash = policy; + EXPECT_THROW(tensogram::streaming_encoder(tmp.path, "{}", opts), + tensogram::encoding_error); + } +} + +TEST(EncodeOptionsTest, StreamingFooterPlacementWritesAFooterHashFrame) { + TempFile tmp; + tensogram::encode_options opts; + opts.aggregate_hash = tensogram::aggregate_hash_policy::footer; + { + tensogram::streaming_encoder enc(tmp.path, "{}", opts); + enc.write_object(kDescriptor, + reinterpret_cast(sample_values().data()), + sample_values().size() * sizeof(float)); + enc.finish(); + } + auto f = tensogram::file::open(tmp.path); + auto raw = f.read_message(0); + auto types = frame_types(raw); + EXPECT_EQ(count_of(types, tensogram::frame_type::footer_hash), 1u); + EXPECT_FALSE(contains(types, tensogram::frame_type::header_hash)); +} + +TEST(EncodeOptionsTest, StreamingAggregateHashNoneOmitsTheFrame) { + TempFile tmp; + tensogram::encode_options opts; + opts.aggregate_hash = tensogram::aggregate_hash_policy::none; + { + tensogram::streaming_encoder enc(tmp.path, "{}", opts); + enc.write_object(kDescriptor, + reinterpret_cast(sample_values().data()), + sample_values().size() * sizeof(float)); + enc.finish(); + } + auto f = tensogram::file::open(tmp.path); + auto raw = f.read_message(0); + auto types = frame_types(raw); + EXPECT_FALSE(contains(types, tensogram::frame_type::footer_hash)); + EXPECT_FALSE(contains(types, tensogram::frame_type::header_hash)); + EXPECT_TRUE(contains(types, tensogram::frame_type::footer_index)); +} + +TEST(EncodeOptionsTest, StreamingHonoursTheCodecBackend) { + TempFile tmp; + tensogram::encode_options opts; + opts.codec_backend = tensogram::compression_backend::ffi; + { + tensogram::streaming_encoder enc(tmp.path, "{}", opts); + enc.write_object(kDescriptor, + reinterpret_cast(sample_values().data()), + sample_values().size() * sizeof(float)); + enc.finish(); + } + auto f = tensogram::file::open(tmp.path); + auto decoded = f.decode_message(0); + ASSERT_EQ(decoded.num_objects(), 1u); + EXPECT_EQ(decoded.object(0).data_as()[2], 3.0f); +} + +// --------------------------------------------------------------------------- +// encode_pre_encoded — the one entry point the C ABI cannot carry these to +// --------------------------------------------------------------------------- + +TEST(EncodeOptionsTest, PreEncodedRejectsOptionsItCannotHonour) { + // There is no `tgm_encode_pre_encoded_with_encode_options`, so the + // placement / backend knobs have no route to the encoder here. Silently + // dropping them would produce a message that contradicts what the caller + // asked for, so this reports the mismatch instead. + const std::string json = + R"({"descriptors":[{"type":"ndarray","ndim":1,"shape":[4],"strides":[4],)" + R"("dtype":"float32","byte_order":"little","encoding":"none","filter":"none",)" + R"("compression":"none"}]})"; + + tensogram::encode_options placement; + placement.aggregate_hash = tensogram::aggregate_hash_policy::footer; + EXPECT_THROW((void)tensogram::encode_pre_encoded(json, sample_objects(), placement), + tensogram::invalid_arg_error); + + tensogram::encode_options backend; + backend.codec_backend = tensogram::compression_backend::pure; + EXPECT_THROW((void)tensogram::encode_pre_encoded(json, sample_objects(), backend), + tensogram::invalid_arg_error); + + // The default options still encode pre-encoded bytes as before. + auto msg = tensogram::encode_pre_encoded(json, sample_objects()); + auto decoded = tensogram::decode(msg.data(), msg.size()); + ASSERT_EQ(decoded.num_objects(), 1u); + EXPECT_EQ(decoded.object(0).data_as()[0], 1.0f); +} + +// --------------------------------------------------------------------------- +// The mask knobs keep working alongside the new fields +// --------------------------------------------------------------------------- + +TEST(EncodeOptionsTest, MaskOptionsCombineWithTheAggregateHashPolicy) { + std::vector values{1.0f, std::numeric_limits::quiet_NaN(), 3.0f, 4.0f}; + std::vector> objects{ + {reinterpret_cast(values.data()), values.size() * sizeof(float)}}; + tensogram::encode_options opts; + opts.allow_nan = true; + opts.aggregate_hash = tensogram::aggregate_hash_policy::footer; + auto msg = tensogram::encode(one_object_json(), objects, opts); + + auto types = frame_types(msg); + EXPECT_TRUE(contains(types, tensogram::frame_type::footer_hash)); + auto decoded = tensogram::decode(msg.data(), msg.size()); + ASSERT_EQ(decoded.num_objects(), 1u); + EXPECT_TRUE(std::isnan(decoded.object(0).data_as()[1])); +} diff --git a/cpp/tests/test_frame_walk.cpp b/cpp/tests/test_frame_walk.cpp new file mode 100644 index 00000000..8086d6f3 --- /dev/null +++ b/cpp/tests/test_frame_walk.cpp @@ -0,0 +1,436 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +// SPDX-License-Identifier: Apache-2.0 +// Copyright 2026 ECMWF +// +// Tests for the lazy frame walker and the typed message header: +// * tensogram::frame_type / frame (tgm_frame_type / TgmFrame) +// * tensogram::frames -> frame_range (tgm_frame_iter_create/_next/_free) +// * tensogram::read_message_header (tgm_message_header) + +#include +#include +#include "test_helpers.hpp" + +#include +#include +#include +#include +#include +#include + +using test_helpers::TempFile; + +namespace { + +// Structural constants from plans/WIRE_FORMAT.md, spelled out here so the +// assertions below read as statements about the wire format rather than +// about the walker that produced them. +constexpr std::size_t kPreambleSize = 24; // §3 +constexpr std::size_t kPostambleSize = 24; // §7 +constexpr std::size_t kFrameHeader = 16; // §2.1 +constexpr std::size_t kFooterCommon = 12; // §2.2 [hash][ENDF] +constexpr std::size_t kFooterNtensor = 20; // §2.2 [cbor_offset][hash][ENDF] +constexpr std::uint16_t kHashPresent = 1u << 1; // §2.5 frame flag bit 1 + +/// One float32 descriptor entry for a 4-element 1-D tensor. +const char* const kDescriptor = + R"({"type":"ndarray","ndim":1,"shape":[4],"strides":[4],"dtype":"float32",)" + R"("byte_order":"little","encoding":"none","filter":"none","compression":"none"})"; + +/// Metadata JSON describing @p n identical float32 objects. +std::string multi_object_json(std::size_t n) { + std::string json = R"({"descriptors":[)"; + for (std::size_t i = 0; i < n; ++i) { + if (i != 0) json += ","; + json += kDescriptor; + } + return json + "]}"; +} + +/// A buffered (random-access) message holding @p n objects: metadata, index +/// and hash frames all live in the header. +std::vector buffered_message(std::size_t n, bool hashed = true) { + const std::vector values{1.0f, 2.0f, 3.0f, 4.0f}; + std::vector> objects; + for (std::size_t i = 0; i < n; ++i) { + objects.emplace_back(reinterpret_cast(values.data()), + values.size() * sizeof(float)); + } + tensogram::encode_options opts; + if (!hashed) opts.hash_algo = ""; + return tensogram::encode(multi_object_json(n), objects, opts); +} + +/// A streaming message: metadata / index / hashes land in the footer and the +/// preamble's `total_length` is never back-filled. +std::vector streamed_message(bool with_preceder) { + TempFile tmp; + const std::vector values{1.0f, 2.0f, 3.0f, 4.0f}; + { + tensogram::streaming_encoder enc(tmp.path, "{}"); + if (with_preceder) enc.write_preceder(R"({"units":"K"})"); + enc.write_object(kDescriptor, + reinterpret_cast(values.data()), + values.size() * sizeof(float)); + enc.finish(); + } + auto f = tensogram::file::open(tmp.path); + return f.read_message(0); +} + +/// Walk @p msg to exhaustion and collect every frame. +std::vector collect(const std::vector& msg) { + std::vector out; + for (const auto& f : tensogram::frames(msg.data(), msg.size())) { + out.push_back(f); + } + return out; +} + +std::vector types_of(const std::vector& frames) { + std::vector types; + types.reserve(frames.size()); + for (const auto& f : frames) types.push_back(f.type()); + return types; +} + +bool contains(const std::vector& types, tensogram::frame_type t) { + return std::find(types.begin(), types.end(), t) != types.end(); +} + +} // namespace + +// --------------------------------------------------------------------------- +// frame_type — the C++ enum carries the wire's frame-type numbers +// --------------------------------------------------------------------------- + +TEST(FrameTypeTest, MirrorsTheWireNumbers) { + using tensogram::frame_type; + EXPECT_EQ(static_cast(frame_type::header_metadata), 1u); + EXPECT_EQ(static_cast(frame_type::header_index), 2u); + EXPECT_EQ(static_cast(frame_type::header_hash), 3u); + // 4 is reserved (the obsolete v2 data-object frame) — no enumerator. + EXPECT_EQ(static_cast(frame_type::footer_hash), 5u); + EXPECT_EQ(static_cast(frame_type::footer_index), 6u); + EXPECT_EQ(static_cast(frame_type::footer_metadata), 7u); + EXPECT_EQ(static_cast(frame_type::preceder_metadata), 8u); + EXPECT_EQ(static_cast(frame_type::ntensor), 9u); +} + +TEST(FrameTypeTest, MatchesTheCEnum) { + using tensogram::frame_type; + EXPECT_EQ(static_cast(frame_type::header_metadata), + static_cast(TGM_FRAME_TYPE_HEADER_METADATA)); + EXPECT_EQ(static_cast(frame_type::footer_metadata), + static_cast(TGM_FRAME_TYPE_FOOTER_METADATA)); + EXPECT_EQ(static_cast(frame_type::ntensor), + static_cast(TGM_FRAME_TYPE_NTENSOR)); +} + +// --------------------------------------------------------------------------- +// frames() — the expected frame sequence +// --------------------------------------------------------------------------- + +TEST(FrameWalkTest, BufferedMessageYieldsHeaderFramesThenOneFramePerObject) { + using tensogram::frame_type; + auto msg = buffered_message(2); + const std::vector expected{ + frame_type::header_metadata, + frame_type::header_index, + frame_type::header_hash, + frame_type::ntensor, + frame_type::ntensor, + }; + EXPECT_EQ(types_of(collect(msg)), expected); +} + +TEST(FrameWalkTest, UnhashedMessageHasNoAggregateHashFrame) { + using tensogram::frame_type; + auto msg = buffered_message(1, /*hashed=*/false); + const std::vector expected{ + frame_type::header_metadata, + frame_type::header_index, + frame_type::ntensor, + }; + EXPECT_EQ(types_of(collect(msg)), expected); +} + +TEST(FrameWalkTest, StreamingMessagePutsMetadataAndIndexInTheFooter) { + using tensogram::frame_type; + auto msg = streamed_message(/*with_preceder=*/false); + const std::vector expected{ + frame_type::header_metadata, + frame_type::ntensor, + frame_type::footer_metadata, + frame_type::footer_hash, + frame_type::footer_index, + }; + EXPECT_EQ(types_of(collect(msg)), expected); +} + +TEST(FrameWalkTest, PrecederMetadataPrecedesItsDataObject) { + using tensogram::frame_type; + auto msg = streamed_message(/*with_preceder=*/true); + auto types = types_of(collect(msg)); + ASSERT_TRUE(contains(types, frame_type::preceder_metadata)); + const auto it = std::find(types.begin(), types.end(), frame_type::preceder_metadata); + ASSERT_NE(it + 1, types.end()); + EXPECT_EQ(*(it + 1), frame_type::ntensor); +} + +TEST(FrameWalkTest, RangeForVisitsEveryFrame) { + auto msg = buffered_message(3); + std::size_t seen = 0; + std::size_t data_objects = 0; + for (const auto& f : tensogram::frames(msg.data(), msg.size())) { + ++seen; + if (f.type() == tensogram::frame_type::ntensor) ++data_objects; + } + EXPECT_EQ(seen, 6u); // metadata + index + hash + 3 data objects + EXPECT_EQ(data_objects, 3u); +} + +TEST(FrameWalkTest, IsDataObjectAgreesWithTheFrameType) { + auto msg = buffered_message(2); + std::size_t data_objects = 0; + for (const auto& f : collect(msg)) { + EXPECT_EQ(f.is_data_object(), f.type() == tensogram::frame_type::ntensor); + if (f.is_data_object()) ++data_objects; + } + EXPECT_EQ(data_objects, 2u); +} + +// --------------------------------------------------------------------------- +// Structural invariants: offsets tile the message and stay in bounds +// --------------------------------------------------------------------------- + +TEST(FrameWalkTest, OffsetsTileTheMessageAndStayInBounds) { + auto msg = buffered_message(2); + std::size_t prev_end = kPreambleSize; + std::size_t walked = 0; + for (const auto& f : collect(msg)) { + EXPECT_GE(f.offset(), prev_end) << "frames must not overlap"; + EXPECT_LE(f.offset() + f.length(), msg.size()) << "frame stays in bounds"; + EXPECT_GE(f.length(), kFrameHeader); + // The whole-frame span always ends on the ENDF sentinel. + const std::size_t end = f.offset() + f.length(); + ASSERT_GE(end, 4u); + EXPECT_EQ(0, std::memcmp(msg.data() + end - 4, "ENDF", 4)) + << "length() must span the frame footer"; + prev_end = end; + ++walked; + } + ASSERT_GT(walked, 0u); + EXPECT_LE(prev_end, msg.size() - kPostambleSize) + << "the postamble is not a frame"; +} + +TEST(FrameWalkTest, FirstFrameStartsAfterThePreamble) { + auto msg = buffered_message(1); + auto frames = collect(msg); + ASSERT_FALSE(frames.empty()); + EXPECT_GE(frames.front().offset(), kPreambleSize); +} + +TEST(FrameWalkTest, PayloadExcludesTheFrameHeaderAndFooter) { + auto msg = buffered_message(1); + auto frames = collect(msg); + ASSERT_FALSE(frames.empty()); + for (const auto& f : frames) { + const std::size_t footer = + f.type() == tensogram::frame_type::ntensor ? kFooterNtensor : kFooterCommon; + EXPECT_EQ(f.payload_size(), f.length() - kFrameHeader - footer); + EXPECT_EQ(f.payload_data(), msg.data() + f.offset() + kFrameHeader); + EXPECT_EQ(f.payload().size(), f.payload_size()); + EXPECT_EQ(static_cast(f.payload().data()), + static_cast(f.payload_data())); + } +} + +TEST(FrameWalkTest, PayloadBorrowsTheCallersBufferAndOutlivesTheRange) { + // Documented lifetime contract: payload() points INTO the caller's message + // buffer, so it stays valid after the range (and its C cursor) are gone. + auto msg = buffered_message(1); + std::vector frames; + { + auto range = tensogram::frames(msg.data(), msg.size()); + for (const auto& f : range) frames.push_back(f); + } // range destroyed here — tgm_frame_iter_free has run + ASSERT_FALSE(frames.empty()); + for (const auto& f : frames) { + ASSERT_GT(f.payload_size(), 0u); + const std::size_t start = f.offset() + kFrameHeader; + EXPECT_EQ(f.payload_data(), msg.data() + start); + EXPECT_EQ(0, std::memcmp(f.payload_data(), msg.data() + start, f.payload_size())) + << "payload must still read the caller's bytes"; + } +} + +TEST(FrameWalkTest, HasHashMirrorsThePerFrameFlagBit) { + auto hashed = buffered_message(1); + auto frames = collect(hashed); + ASSERT_FALSE(frames.empty()); + for (const auto& f : frames) { + EXPECT_TRUE(f.has_hash()); + EXPECT_EQ(f.has_hash(), (f.flags() & kHashPresent) != 0); + EXPECT_EQ(f.version(), 1u); + } + + auto plain = buffered_message(1, /*hashed=*/false); + for (const auto& f : collect(plain)) { + EXPECT_FALSE(f.has_hash()); + EXPECT_EQ(f.has_hash(), (f.flags() & kHashPresent) != 0); + } +} + +// --------------------------------------------------------------------------- +// End vs malformed: a clean end stops, a broken chain throws +// --------------------------------------------------------------------------- + +TEST(FrameWalkTest, CleanEndDoesNotThrowAndStaysAtEnd) { + auto msg = buffered_message(1); + auto range = tensogram::frames(msg.data(), msg.size()); + auto it = range.begin(); + std::size_t n = 0; + for (; it != range.end(); ++it) ++n; + EXPECT_GT(n, 0u); + EXPECT_TRUE(it == range.end()); + // Advancing an exhausted cursor stays safely at the end. + EXPECT_NO_THROW(++it); + EXPECT_TRUE(it == range.end()); +} + +TEST(FrameWalkTest, IteratorsSurviveAMoveOfTheRange) { + // The range owns the C cursor through a unique_ptr, so a move transfers + // the same pointer value — an iterator taken before the move keeps + // walking the same cursor. + auto msg = buffered_message(2); + const std::size_t total = collect(msg).size(); + + auto range = tensogram::frames(msg.data(), msg.size()); + auto it = range.begin(); + ASSERT_TRUE(it != range.end()); + const auto first_type = it->type(); + + auto moved = std::move(range); + std::size_t seen = 1; // the frame already pulled above + std::size_t prev_offset = it->offset(); + for (++it; it != moved.end(); ++it) { + EXPECT_GT(it->offset(), prev_offset) << "the walk keeps moving forward"; + prev_offset = it->offset(); + ++seen; + } + EXPECT_EQ(seen, total) << "the moved-into range finishes the same walk"; + EXPECT_EQ(first_type, tensogram::frame_type::header_metadata); +} + +TEST(FrameWalkTest, TruncatedFrameChainThrowsAfterYieldingTheIntactFrames) { + auto msg = buffered_message(2); + auto frames = collect(msg); + ASSERT_FALSE(frames.empty()); + // Cut *inside* the last frame: the preamble still parses, the chain does not. + const std::size_t cut = frames.back().offset() + 8; + std::vector truncated(msg.begin(), msg.begin() + static_cast(cut)); + + std::size_t yielded = 0; + EXPECT_THROW( + { + for (const auto& f : tensogram::frames(truncated.data(), truncated.size())) { + (void)f; + ++yielded; + } + }, + tensogram::framing_error); + EXPECT_GT(yielded, 0u) << "the frames before the cut are still yielded"; +} + +TEST(FrameWalkTest, NonMessageBufferThrows) { + const std::string junk = "not a tensogram message at all!!!!!!!!!!"; + EXPECT_THROW((void)tensogram::frames(reinterpret_cast(junk.data()), + junk.size()), + tensogram::framing_error); +} + +TEST(FrameWalkTest, TruncatedPreambleThrows) { + auto msg = buffered_message(1); + EXPECT_THROW((void)tensogram::frames(msg.data(), 8), tensogram::framing_error); +} + +TEST(FrameWalkTest, NullBufferIsRejected) { + EXPECT_THROW((void)tensogram::frames(nullptr, 0), tensogram::invalid_arg_error); +} + +// --------------------------------------------------------------------------- +// read_message_header +// --------------------------------------------------------------------------- + +TEST(MessageHeaderTest, BufferedPreambleDescribesARandomAccessMessage) { + auto msg = buffered_message(2); + const auto h = tensogram::read_message_header(msg.data(), msg.size()); + EXPECT_EQ(h.version(), tensogram::wire_version); + EXPECT_EQ(h.total_length(), msg.size()); + EXPECT_TRUE(h.has_header_metadata()); + EXPECT_TRUE(h.has_header_index()); + EXPECT_FALSE(h.has_footer_metadata()); + EXPECT_FALSE(h.has_footer_index()); + EXPECT_FALSE(h.has_preceder_metadata()); + EXPECT_TRUE(h.has_hashes_present()) << "the default encode hashes every frame"; +} + +TEST(MessageHeaderTest, BufferedFlagsMatchTheFramesExactly) { + auto msg = buffered_message(2); + const auto h = tensogram::read_message_header(msg.data(), msg.size()); + const auto types = types_of(collect(msg)); + using tensogram::frame_type; + EXPECT_EQ(h.has_header_metadata(), contains(types, frame_type::header_metadata)); + EXPECT_EQ(h.has_footer_metadata(), contains(types, frame_type::footer_metadata)); + EXPECT_EQ(h.has_header_index(), contains(types, frame_type::header_index)); + EXPECT_EQ(h.has_footer_index(), contains(types, frame_type::footer_index)); + EXPECT_EQ(h.has_header_hashes(), contains(types, frame_type::header_hash)); + EXPECT_EQ(h.has_footer_hashes(), contains(types, frame_type::footer_hash)); + EXPECT_EQ(h.has_preceder_metadata(), contains(types, frame_type::preceder_metadata)); +} + +TEST(MessageHeaderTest, StreamingFlagsNeverUnderstateTheFramesPresent) { + // The streaming encoder writes its preamble before it knows what follows, + // so PRECEDER_METADATA is set advisorily: only "frame present => flag set" + // holds. + for (bool with_preceder : {false, true}) { + auto msg = streamed_message(with_preceder); + const auto h = tensogram::read_message_header(msg.data(), msg.size()); + const auto types = types_of(collect(msg)); + using tensogram::frame_type; + const auto implies = [&](frame_type t, bool flag) { return !contains(types, t) || flag; }; + EXPECT_TRUE(implies(frame_type::header_metadata, h.has_header_metadata())); + EXPECT_TRUE(implies(frame_type::footer_metadata, h.has_footer_metadata())); + EXPECT_TRUE(implies(frame_type::header_index, h.has_header_index())); + EXPECT_TRUE(implies(frame_type::footer_index, h.has_footer_index())); + EXPECT_TRUE(implies(frame_type::header_hash, h.has_header_hashes())); + EXPECT_TRUE(implies(frame_type::footer_hash, h.has_footer_hashes())); + EXPECT_TRUE(implies(frame_type::preceder_metadata, h.has_preceder_metadata())); + // Streaming never back-fills the length through the non-seeking sink. + EXPECT_EQ(h.total_length(), 0u); + EXPECT_TRUE(h.has_footer_metadata()); + EXPECT_TRUE(h.has_footer_index()); + } +} + +TEST(MessageHeaderTest, RejectsBuffersThatAreNotMessages) { + const std::string junk = "not a tensogram message at all!!!!!!!!!!"; + EXPECT_THROW((void)tensogram::read_message_header( + reinterpret_cast(junk.data()), junk.size()), + tensogram::framing_error); + + auto msg = buffered_message(1); + EXPECT_THROW((void)tensogram::read_message_header(msg.data(), 8), + tensogram::framing_error); + EXPECT_THROW((void)tensogram::read_message_header(nullptr, 0), + tensogram::invalid_arg_error); +} diff --git a/cpp/tests/test_remote.cpp b/cpp/tests/test_remote.cpp new file mode 100644 index 00000000..cd7b74d3 --- /dev/null +++ b/cpp/tests/test_remote.cpp @@ -0,0 +1,201 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +// SPDX-License-Identifier: Apache-2.0 +// Copyright 2026 ECMWF +// +// Tests for the synchronous remote surface: +// * tensogram::is_remote_url (tgm_is_remote_url) +// * tensogram::remote_scan_options (TgmRemoteScanOptions) +// * tensogram::file::open_remote (tgm_file_open_remote) +// +// The remote backend is an opt-in Cargo feature. Symbols always link, so +// every build exercises the argument validation and the local-path answers; +// the tests that need a working backend are compiled only when the C API was +// built with `--features remote` (CMake: -DTENSOGRAM_REMOTE=ON), and the +// feature-off build asserts the honest "this build cannot" behaviour instead. + +#include +#include +#include "test_helpers.hpp" + +#include +#include +#include +#include +#include + +using test_helpers::TempFile; + +#ifdef TENSOGRAM_REMOTE +namespace { + +/// Write a two-message `.tgm` and return the `file://` URL addressing it. +/// +/// `file://` travels the same object-store code path as `s3://` & friends, +/// so it exercises the real remote backend without a network. +std::string write_fixture(const std::string& path) { + const std::vector a{1.0f, 2.0f, 3.0f, 4.0f}; + const std::vector b{5.0f, 6.0f, 7.0f, 8.0f}; + auto f = tensogram::file::create(path); + f.append_raw(test_helpers::encode_simple_f32(a)); + f.append_raw(test_helpers::encode_simple_f32(b)); + return "file://" + path; +} + +} // namespace +#endif // TENSOGRAM_REMOTE + +// --------------------------------------------------------------------------- +// remote_scan_options +// --------------------------------------------------------------------------- + +TEST(RemoteTest, ScanOptionsDefaultToTheBidirectionalWalk) { + const tensogram::remote_scan_options opts; + EXPECT_TRUE(opts.bidirectional); +} + +// --------------------------------------------------------------------------- +// is_remote_url +// --------------------------------------------------------------------------- + +TEST(RemoteTest, LocalSourcesAreNeverRemote) { + // True in every build: these belong to the local backend (file::open). + EXPECT_FALSE(tensogram::is_remote_url("/tmp/data.tgm")); + EXPECT_FALSE(tensogram::is_remote_url("data.tgm")); + EXPECT_FALSE(tensogram::is_remote_url("./relative/data.tgm")); + EXPECT_FALSE(tensogram::is_remote_url("file:///tmp/data.tgm")); + EXPECT_FALSE(tensogram::is_remote_url("")); + EXPECT_FALSE(tensogram::is_remote_url("ftp://host/data.tgm")); +} + +TEST(RemoteTest, ObjectStoreSchemesMatchThisBuildsCapability) { + // A remote-capable build recognises every object-store scheme; a build + // without the feature genuinely cannot open any of them, so "not remote + // for me" is the honest answer (the C layer records why). +#ifdef TENSOGRAM_REMOTE + const bool expected = true; +#else + const bool expected = false; +#endif + for (const char* url : {"s3://bucket/key.tgm", "s3a://bucket/key.tgm", + "gs://bucket/key.tgm", "az://container/key.tgm", + "azure://container/key.tgm", "http://host/key.tgm", + "https://host/key.tgm"}) { + EXPECT_EQ(tensogram::is_remote_url(url), expected) << url; + } +} + +TEST(RemoteTest, SchemesAreComparedCaseInsensitively) { +#ifdef TENSOGRAM_REMOTE + EXPECT_TRUE(tensogram::is_remote_url("S3://bucket/key.tgm")); + EXPECT_TRUE(tensogram::is_remote_url("HTTPS://host/key.tgm")); +#else + EXPECT_FALSE(tensogram::is_remote_url("S3://bucket/key.tgm")); +#endif +} + +TEST(RemoteTest, IsRemoteUrlAcceptsAStringViewOfASubstring) { + // The predicate takes a string_view, so it must not rely on the caller's + // buffer being NUL-terminated at the end of the view. + const std::string haystack = "s3://bucket/key.tgmXXXX"; + const std::string_view view(haystack.data(), haystack.size() - 4); +#ifdef TENSOGRAM_REMOTE + EXPECT_TRUE(tensogram::is_remote_url(view)); +#else + EXPECT_FALSE(tensogram::is_remote_url(view)); +#endif +} + +// --------------------------------------------------------------------------- +// file::open_remote — argument validation (identical in every build) +// --------------------------------------------------------------------------- + +TEST(RemoteTest, OpenRemoteRejectsAnEmbeddedNulInTheSource) { + // A C string cannot carry an interior NUL: silently truncating would open + // a *different* object than the caller asked for. + const std::string source("s3://bucket/key.tgm\0extra", 25); + EXPECT_THROW((void)tensogram::file::open_remote(source), tensogram::invalid_arg_error); +} + +TEST(RemoteTest, OpenRemoteRejectsEmbeddedNulsInStorageOptions) { + const std::map bad_key{ + {std::string("aws_region\0x", 12), "eu-west-1"}}; + EXPECT_THROW((void)tensogram::file::open_remote("s3://bucket/key.tgm", bad_key), + tensogram::invalid_arg_error); + + const std::map bad_value{ + {"aws_region", std::string("eu-west-1\0x", 11)}}; + EXPECT_THROW((void)tensogram::file::open_remote("s3://bucket/key.tgm", bad_value), + tensogram::invalid_arg_error); +} + +// --------------------------------------------------------------------------- +// file::open_remote — behaviour, per build flavour +// --------------------------------------------------------------------------- + +#ifndef TENSOGRAM_REMOTE + +TEST(RemoteTest, OpenRemoteWithoutTheFeatureExplainsHowToEnableIt) { + try { + (void)tensogram::file::open_remote("s3://bucket/key.tgm"); + FAIL() << "expected a remote_error from a build without the feature"; + } catch (const tensogram::remote_error& e) { + EXPECT_EQ(e.code(), TGM_ERROR_REMOTE); + const std::string what = e.what(); + EXPECT_NE(what.find("remote"), std::string::npos) << what; + } +} + +TEST(RemoteTest, OpenRemoteWithoutTheFeatureStillValidatesArgumentsFirst) { + // Argument validation runs before the feature check, so a mistake is + // reported as a mistake rather than as "no remote support". + const std::string source("s3://bucket/key.tgm\0extra", 25); + EXPECT_THROW((void)tensogram::file::open_remote(source), tensogram::invalid_arg_error); +} + +#else // TENSOGRAM_REMOTE + +TEST(RemoteTest, OpenRemoteReadsAFileUrlThroughTheOrdinaryFileApi) { + TempFile tmp; + const std::string url = write_fixture(tmp.path); + + auto remote = tensogram::file::open_remote(url); + ASSERT_EQ(remote.message_count(), 2u); + EXPECT_EQ(remote.path(), url); + + auto raw = remote.read_message(0); + EXPECT_FALSE(raw.empty()); + auto decoded = remote.decode_message(1); + ASSERT_EQ(decoded.num_objects(), 1u); + EXPECT_EQ(decoded.object(0).data_as()[0], 5.0f); + + // The frame walker works on remotely-read bytes like any other message. + const auto header = tensogram::read_message_header(raw.data(), raw.size()); + EXPECT_EQ(header.version(), tensogram::wire_version); +} + +TEST(RemoteTest, OpenRemoteAcceptsForwardOnlyScanOptions) { + TempFile tmp; + const std::string url = write_fixture(tmp.path); + tensogram::remote_scan_options opts; + opts.bidirectional = false; + auto remote = tensogram::file::open_remote(url, {}, opts); + EXPECT_EQ(remote.message_count(), 2u); +} + +TEST(RemoteTest, OpenRemoteReportsAMissingObjectAsARemoteError) { + EXPECT_THROW((void)tensogram::file::open_remote("file:///nonexistent/missing.tgm"), + tensogram::remote_error); +} + +TEST(RemoteTest, OpenRemoteRejectsAnUnparseableUrl) { + EXPECT_THROW((void)tensogram::file::open_remote("s3://"), tensogram::remote_error); +} + +#endif // TENSOGRAM_REMOTE diff --git a/cpp/tests/test_typed_enums.cpp b/cpp/tests/test_typed_enums.cpp new file mode 100644 index 00000000..87f79fe9 --- /dev/null +++ b/cpp/tests/test_typed_enums.cpp @@ -0,0 +1,180 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +// SPDX-License-Identifier: Apache-2.0 +// Copyright 2026 ECMWF +// +// Tests for the typed descriptor enums: +// * tensogram::dtype + decoded_object::dtype_enum() (tgm_object_dtype_enum) +// * tensogram::byte_order + decoded_object::byte_order_enum() +// (tgm_object_byte_order_enum) +// +// The paired string getters (dtype_string() / byte_order_string()) stay the +// unambiguous bounds check: they return "" for exactly the inputs on which the +// C enum accessors fall back to their zero variant. + +#include +#include +#include "test_helpers.hpp" + +#include +#include +#include +#include + +namespace { + +/// Encode one 1-D object of @p dtype_name, then decode it. +/// +/// @p elem_bits is the element width in bits (1 for the sub-byte bitmask +/// dtype), @p count the number of elements. Decoding keeps the wire byte +/// order so that a big-endian object is not silently swapped on the way out. +std::vector encode_one(const std::string& dtype_name, + std::size_t elem_bits, + std::size_t count, + const std::string& byte_order_name) { + const std::size_t stride = elem_bits >= 8 ? elem_bits / 8 : 1; + const std::size_t bytes = (elem_bits * count + 7) / 8; + // encode() consumes the bytes during the call, so a local buffer is fine. + const std::vector data(bytes, 0x01); + const std::string json = + R"({"descriptors":[{"type":"ndarray","ndim":1,"shape":[)" + std::to_string(count) + + R"(],"strides":[)" + std::to_string(stride) + R"(],"dtype":")" + dtype_name + + R"(","byte_order":")" + byte_order_name + + R"(","encoding":"none","filter":"none","compression":"none"}]})"; + std::vector> objects{{data.data(), data.size()}}; + return tensogram::encode(json, objects); +} + +struct dtype_case { + const char* name; + std::size_t elem_bits; + tensogram::dtype expected; +}; + +/// Every dtype the encoder accepts, paired with the C++ enumerator it must +/// map to. A new core dtype without a C++ enumerator fails to compile here. +const std::vector& all_dtypes() { + using tensogram::dtype; + static const std::vector cases{ + {"float16", 16, dtype::float16}, + {"bfloat16", 16, dtype::bfloat16}, + {"float32", 32, dtype::float32}, + {"float64", 64, dtype::float64}, + {"complex64", 64, dtype::complex64}, + {"complex128", 128, dtype::complex128}, + {"int8", 8, dtype::int8}, + {"int16", 16, dtype::int16}, + {"int32", 32, dtype::int32}, + {"int64", 64, dtype::int64}, + {"uint8", 8, dtype::uint8}, + {"uint16", 16, dtype::uint16}, + {"uint32", 32, dtype::uint32}, + {"uint64", 64, dtype::uint64}, + {"bitmask", 1, dtype::bitmask}, + }; + return cases; +} + +} // namespace + +// --------------------------------------------------------------------------- +// dtype +// --------------------------------------------------------------------------- + +TEST(DtypeEnumTest, MirrorsTheCEnumValues) { + using tensogram::dtype; + EXPECT_EQ(static_cast(dtype::float16), static_cast(TGM_DTYPE_FLOAT16)); + EXPECT_EQ(static_cast(dtype::bfloat16), static_cast(TGM_DTYPE_BFLOAT16)); + EXPECT_EQ(static_cast(dtype::float32), static_cast(TGM_DTYPE_FLOAT32)); + EXPECT_EQ(static_cast(dtype::float64), static_cast(TGM_DTYPE_FLOAT64)); + EXPECT_EQ(static_cast(dtype::complex64), static_cast(TGM_DTYPE_COMPLEX64)); + EXPECT_EQ(static_cast(dtype::complex128), static_cast(TGM_DTYPE_COMPLEX128)); + EXPECT_EQ(static_cast(dtype::int8), static_cast(TGM_DTYPE_INT8)); + EXPECT_EQ(static_cast(dtype::int16), static_cast(TGM_DTYPE_INT16)); + EXPECT_EQ(static_cast(dtype::int32), static_cast(TGM_DTYPE_INT32)); + EXPECT_EQ(static_cast(dtype::int64), static_cast(TGM_DTYPE_INT64)); + EXPECT_EQ(static_cast(dtype::uint8), static_cast(TGM_DTYPE_UINT8)); + EXPECT_EQ(static_cast(dtype::uint16), static_cast(TGM_DTYPE_UINT16)); + EXPECT_EQ(static_cast(dtype::uint32), static_cast(TGM_DTYPE_UINT32)); + EXPECT_EQ(static_cast(dtype::uint64), static_cast(TGM_DTYPE_UINT64)); + EXPECT_EQ(static_cast(dtype::bitmask), static_cast(TGM_DTYPE_BITMASK)); +} + +TEST(DtypeEnumTest, AgreesWithTheStringGetterForEveryDtype) { + tensogram::decode_options opts; + opts.native_byte_order = false; // keep the wire dtype layout untouched + for (const auto& c : all_dtypes()) { + auto encoded = encode_one(c.name, c.elem_bits, 8, "little"); + auto msg = tensogram::decode(encoded.data(), encoded.size(), opts); + ASSERT_EQ(msg.num_objects(), 1u) << c.name; + const auto obj = msg.object(0); + EXPECT_EQ(obj.dtype_string(), c.name); + EXPECT_EQ(obj.dtype_enum(), c.expected) << c.name; + } +} + +TEST(DtypeEnumTest, IsUsableInASwitch) { + // The point of the typed accessor: switch instead of strcmp. + auto encoded = encode_one("int32", 32, 4, "little"); + auto msg = tensogram::decode(encoded.data(), encoded.size()); + std::size_t width = 0; + switch (msg.object(0).dtype_enum()) { + case tensogram::dtype::int32: width = 4; break; + default: width = 0; break; + } + EXPECT_EQ(width, 4u); +} + +TEST(DtypeEnumTest, OutOfRangeIndexThrowsInsteadOfReturningTheZeroVariant) { + auto encoded = encode_one("float64", 64, 4, "little"); + auto msg = tensogram::decode(encoded.data(), encoded.size()); + ASSERT_EQ(msg.num_objects(), 1u); + // The C accessor has no spare code for failure — it returns the zero + // variant (float16) and records the reason. The C++ wrapper bounds-checks + // through the paired string getter, which returns "" for exactly those + // inputs, and raises instead of handing back a plausible-looking dtype. + EXPECT_EQ(msg.object(1).dtype_string(), ""); + EXPECT_THROW((void)msg.object(1).dtype_enum(), tensogram::invalid_arg_error); + EXPECT_THROW((void)msg.object(99).dtype_enum(), tensogram::invalid_arg_error); +} + +// --------------------------------------------------------------------------- +// byte_order +// --------------------------------------------------------------------------- + +TEST(ByteOrderEnumTest, MirrorsTheCEnumValues) { + EXPECT_EQ(static_cast(tensogram::byte_order::little), + static_cast(TGM_BYTE_ORDER_LITTLE)); + EXPECT_EQ(static_cast(tensogram::byte_order::big), + static_cast(TGM_BYTE_ORDER_BIG)); +} + +TEST(ByteOrderEnumTest, AgreesWithTheStringGetter) { + tensogram::decode_options opts; + opts.native_byte_order = false; + const std::pair cases[]{ + {"little", tensogram::byte_order::little}, + {"big", tensogram::byte_order::big}, + }; + for (const auto& [name, expected] : cases) { + auto encoded = encode_one("int16", 16, 4, name); + auto msg = tensogram::decode(encoded.data(), encoded.size(), opts); + ASSERT_EQ(msg.num_objects(), 1u); + const auto obj = msg.object(0); + EXPECT_EQ(obj.byte_order_string(), name); + EXPECT_EQ(obj.byte_order_enum(), expected) << name; + } +} + +TEST(ByteOrderEnumTest, OutOfRangeIndexThrows) { + auto encoded = encode_one("uint8", 8, 4, "little"); + auto msg = tensogram::decode(encoded.data(), encoded.size()); + EXPECT_EQ(msg.object(1).byte_order_string(), ""); + EXPECT_THROW((void)msg.object(1).byte_order_enum(), tensogram::invalid_arg_error); +} diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index a31ee070..f14bd72a 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -35,6 +35,7 @@ - [Remote Access (S3, GCS, Azure, HTTP)](guide/remote-access.md) - [Iterators](guide/iterators.md) - [Reading Metadata](guide/metadata.md) +- [Frame Introspection](guide/frame-introspection.md) - [Python API](guide/python-api.md) - [C API](guide/c-api.md) - [C++ API](guide/cpp-api.md) diff --git a/docs/src/guide/c-api.md b/docs/src/guide/c-api.md index 2f212261..87bfe4a0 100644 --- a/docs/src/guide/c-api.md +++ b/docs/src/guide/c-api.md @@ -2,10 +2,12 @@ Tensogram exposes a flat C ABI through the `tensogram-ffi` crate. The generated header is `tensogram.h`; all public functions are prefixed -`tgm_`, most public types follow the `tgm_*_t` pattern (the option -structs `TgmEncodeMaskOptions` and `TgmDecodeMaskOptions` are PascalCase -exceptions inherited from the underlying Rust types), and all error -codes are members of the `tgm_error` enum. +`tgm_`, most public types follow the `tgm_*_t` pattern (the PODs +`TgmEncodeMaskOptions`, `TgmDecodeMaskOptions`, `TgmEncodeOptions`, +`TgmScanOptions`, `TgmRemoteScanOptions`, `TgmFrame`, and +`TgmMessageHeader` are PascalCase exceptions inherited from the +underlying Rust types), and all error codes are members of the +`tgm_error` enum. The C++ wrapper at `cpp/include/tensogram.hpp` is built directly on top of this C API; see [C++ API](cpp-api.md) for the higher-level @@ -214,6 +216,163 @@ int main(void) { } ``` +## Synchronous remote access (opt-in `remote` feature) + +A remote `.tgm` (S3, GCS, Azure, HTTP) can be read through the ordinary +**blocking** file API — no async ABI required: + +```c +const char *keys[] = { "aws_region", "aws_access_key_id" }; +const char *values[] = { "eu-west-1", "AKIA..." }; +TgmRemoteScanOptions opts = { .bidirectional = true }; /* NULL = defaults */ + +tgm_file_t *file = NULL; +tgm_error rc = tgm_file_open_remote("s3://bucket/forecast.tgm", + keys, values, 2, &opts, &file); +if (rc != TGM_ERROR_OK) { + fprintf(stderr, "open_remote failed: %s\n", tgm_last_error()); + return 1; +} + +/* `file` is a PLAIN tgm_file_t — every existing file function works. */ +size_t n = 0; +tgm_file_message_count(file, &n); +tgm_file_close(file); +``` + +`keys` / `values` are parallel arrays of `n_options` backend storage +options (credentials, region, endpoint, …) forwarded verbatim to the +object-store backend; pass `0` / `NULL` for none. `tgm_is_remote_url` +tells the two backends apart before you open anything — the recognised +schemes are `s3`, `s3a`, `gs`, `az`, `azure`, `http`, `https` (compared +case-insensitively). Plain paths and `file://` URLs are **not** remote; +they belong to `tgm_file_open`. + +> **This is an opt-in Cargo feature and is NOT in the published +> tarballs.** The prebuilt `tensogram-ffi--.tar.gz` +> assets are built with default features, exactly like the existing +> `async` surface. To get a library that can actually open a remote +> source, build it yourself: +> +> ```bash +> cargo cinstall --release -p tensogram-ffi --features=remote \ +> --prefix="$HOME/.local" --libdir=lib +> # in-tree / C++ wrapper builds: +> cargo build --release -p tensogram-ffi --features=remote +> cmake -S cpp -B build -DTENSOGRAM_REMOTE=ON # default OFF +> ``` +> +> Both symbols are exported **either way**, so you never hit an +> undefined symbol at link time. In a build without the feature: +> +> - `tgm_is_remote_url` returns `false` for *every* input — such a build +> genuinely cannot open any URL, so "not remote for me" is the honest +> answer; +> - `tgm_file_open_remote` returns `TGM_ERROR_REMOTE` with a +> `tgm_last_error()` message naming `--features=remote`. +> +> Argument validation (NULL `source` / `out`, mismatched option arrays) +> runs **before** the feature check, so a genuine mistake is reported as +> `TGM_ERROR_INVALID_ARG` in both builds. + +The `remote` feature is independent of `async` — neither implies the +other. For the async equivalent see +[C++ Async API](cpp-async.md#remote-reads-open_remote); for the transport +behaviour (request budget, bidirectional scan, limitations) see +[Remote Access](remote-access.md). + +## Frame introspection + +`tgm_message_header` reads a message's 24-byte envelope into a +`TgmMessageHeader` POD (wire version, `total_length`, and eight `has_*` +booleans), and `tgm_frame_iter_create` / `_next` / `_free` walk that +message's frames, filling a caller-provided `TgmFrame` per step: + +```c +TgmMessageHeader h; +tgm_message_header(msg, msg_len, &h); /* 24 bytes; walks nothing */ + +tgm_frame_iter_t *it = tgm_frame_iter_create(msg, msg_len); +TgmFrame f; +while (tgm_frame_iter_next(it, &f)) { + /* f.frame_type is a tgm_frame_type; f.payload borrows msg */ + if (tgm_frame_has_hash(&f)) { /* the inline slot holds a digest */ } +} +const char *err = tgm_last_error(); /* NULL => clean end, else framing */ +tgm_frame_iter_free(it); +``` + +`f.payload` / `f.payload_len` are the frame **content** (the 16-byte +frame header and the type-specific footer are stripped) and point +**into** `msg`: nothing to free, valid for as long as `msg` lives, and +unaffected by later `_next` calls or by `tgm_frame_iter_free`. The +iterator borrows `msg` for its whole lifetime, so `msg` must outlive it. + +Both entry points take **one** message — use `tgm_scan` to locate the +boundaries in a multi-message buffer and pass each slice. The full +contract (frame types, offsets, payload boundaries, the streaming-flag +nuance) is in [Frame Introspection](frame-introspection.md). + +## Typed enums and the full encode options + +Four new enums make descriptor and option values `switch`-able instead of +`strcmp`-able. Each one's **zero value is the library default**, so a +zero-initialised option struct asks for exactly the previous behaviour: + +| Enum | Variants | Notes | +|---|---|---| +| `tgm_dtype` | `TGM_DTYPE_FLOAT16` … `TGM_DTYPE_BITMASK` (15) | FFI convenience — the wire stores the dtype as a *string* | +| `tgm_byte_order` | `TGM_BYTE_ORDER_LITTLE` / `_BIG` | likewise a string on the wire | +| `tgm_aggregate_hash_policy` | `AUTO` / `NONE` / `HEADER` / `FOOTER` / `BOTH` | where the aggregate hash frame goes | +| `tgm_compression_backend` | `AUTO` / `FFI` / `PURE` | which szip / zstd implementation to use | + +`tgm_frame_type` (above) follows the same pattern, except that *its* +numbers really are wire values. + +Two typed accessors join the existing string getters, which are +unchanged and still supported: + +```c +switch (tgm_object_dtype_enum(msg, 0)) { /* vs tgm_object_dtype */ + case TGM_DTYPE_FLOAT32: /* 4 bytes per element */ break; + default: break; +} +tgm_byte_order bo = tgm_object_byte_order_enum(msg, 0); +``` + +An enum return has no spare code for failure: a NULL handle or an +out-of-range index records the reason in `tgm_last_error()` and returns +the **zero variant** (`TGM_DTYPE_FLOAT16`, `TGM_BYTE_ORDER_LITTLE`). To +bounds-check unambiguously, compare the index against +`tgm_message_num_objects`, or call the paired string getter, which +returns `NULL` for exactly the same inputs. + +`TgmEncodeOptions` supersedes `TgmEncodeMaskOptions` (which stays for +source compatibility): it carries the same six mask fields **plus** the +three knobs that previously had no C surface at all — the hash algorithm, +the aggregate-hash placement, and the codec backend: + +```c +TgmEncodeOptions opts = {0}; /* every zero IS the default */ +opts.hash = "xxh3"; /* NULL / omitted = no hashing */ +opts.aggregate_hash = TGM_AGGREGATE_HASH_POLICY_BOTH; +opts.compression_backend = TGM_COMPRESSION_BACKEND_PURE; + +tgm_bytes_t enc = {0}; +tgm_encode_with_encode_options(meta_json, ptrs, lens, 1, 0, &opts, &enc); +``` + +There is one `*_with_encode_options` entry point per encode target — +`tgm_encode_with_encode_options`, +`tgm_file_append_with_encode_options`, and +`tgm_streaming_encoder_create_with_encode_options` — and passing `NULL` +options means the library defaults. Note that the algorithm name lives +in `options->hash`, so these functions have no separate `hash_algo` +argument. `TGM_AGGREGATE_HASH_POLICY_HEADER` and `..._BOTH` are +**buffered-mode only**: the streaming constructor rejects them with +`TGM_ERROR_ENCODING`, because a streaming writer emits its header before +any data object exists. + ## Memory ownership - Handles returned by `tgm_*` constructors (e.g. `tgm_decode` returns @@ -222,6 +381,9 @@ int main(void) { - Pointers returned by accessor functions (e.g. `tgm_object_data`, `tgm_object_shape`) are *borrowed* from the parent handle and are valid only until the parent is freed. +- `TgmFrame::payload` borrows the caller's *message buffer*, not the + iterator: it stays valid for as long as that buffer lives, and + `tgm_frame_iter_free` does not invalidate it. - `tgm_bytes_t` returned by encode functions must be freed with `tgm_bytes_free`. - `tgm_last_error()` returns a thread-local pointer to the most recent @@ -275,6 +437,10 @@ release for now. The Rust API has the same caveat. ## See also - [C++ API](cpp-api.md) — RAII / exceptions wrapper over the C API. +- [Frame Introspection](frame-introspection.md) — the language-neutral + contract behind `tgm_frame_iter_*` / `tgm_message_header`. +- [Remote Access](remote-access.md) — transport behaviour of the + object-store backend behind `tgm_file_open_remote`. - [Error Handling](error-handling.md) — taxonomy of `tgm_error` variants and what each one means. - [Internals](../internals.md) — wire format, encoding pipeline. diff --git a/docs/src/guide/cpp-api.md b/docs/src/guide/cpp-api.md index 16ced304..07c88782 100644 --- a/docs/src/guide/cpp-api.md +++ b/docs/src/guide/cpp-api.md @@ -46,6 +46,7 @@ const float* values = obj.data_as(); | `buffer_iterator` | `tgm_buffer_iter_t` | `tgm_buffer_iter_free` | | `file_iterator` | `tgm_file_iter_t` | `tgm_file_iter_free` | | `object_iterator` | `tgm_object_iter_t` | `tgm_object_iter_free` | +| `frame_range` | `tgm_frame_iter_t` | `tgm_frame_iter_free` | | `streaming_encoder` | `tgm_streaming_encoder_t` | `tgm_streaming_encoder_free` | All classes are move-only (copy deleted). Handles are released automatically when the object goes out of scope. @@ -93,6 +94,71 @@ An invalid level string or a missing file throws `tensogram::invalid_arg_error` See [Iterators](iterators.md#c-api) for buffer, file, and object iterator usage. +## Frame walker and message header + +`tensogram::frames()` returns a lazy, move-only `frame_range` over the frames of **one** message, usable in a range-for; `tensogram::read_message_header()` decodes that message's 24-byte envelope into a `message_header` value: + +```cpp +const auto h = tensogram::read_message_header(msg.data(), msg.size()); +if (h.has_header_index()) { /* random-access layout */ } + +for (const auto& f : tensogram::frames(msg.data(), msg.size())) { + if (f.is_data_object()) { // frame_type::ntensor + std::string_view content = f.payload(); // borrows `msg` + use(f.offset(), f.length(), f.has_hash(), content); + } +} +``` + +`frame::payload()` / `payload_data()` view the caller's buffer — nothing to free, and the views stay valid after the range is destroyed, for as long as the buffer lives. A clean end simply terminates the loop; a truncated or inconsistent frame chain throws `tensogram::framing_error` from the increment that discovers it, after the intact frames have been yielded. The full cross-language contract — offsets, payload boundaries, and why a streaming message's header flags are advisory — is in [Frame Introspection](frame-introspection.md). + +## Synchronous remote reads + +`tensogram::file::open_remote()` opens an S3 / GCS / Azure / HTTP `.tgm` through the ordinary blocking API. The result is a plain `file`, so `message_count()`, `read_message()`, `decode_message()`, `file_iterator`, … all work unchanged: + +```cpp +if (tensogram::is_remote_url(source)) { + auto f = tensogram::file::open_remote( + source, + {{"aws_region", "eu-west-1"}}, // storage options + tensogram::remote_scan_options{}); // bidirectional by default + auto msg = f.decode_message(0); +} else { + auto f = tensogram::file::open(source); +} +``` + +Requires the FFI built with `--features=remote` (`cmake -S cpp -B build -DTENSOGRAM_REMOTE=ON`; the default is **OFF**, and the option is independent of `TENSOGRAM_ASYNC` / `TENSOGRAM_ASYNC_REMOTE`). The symbols always link: in a build without the feature `is_remote_url` returns `false` for every input and `open_remote` throws `tensogram::remote_error` with a message naming the missing feature. See the [C API notes on the `remote` feature](c-api.md#synchronous-remote-access-opt-in-remote-feature) — in particular, it is **not** in the published C-API tarballs. + +## Typed enums and encode options + +`enum class dtype`, `byte_order`, `aggregate_hash_policy`, and `compression_backend` mirror the C enums, and each one's default is the library default. `decoded_object` gained the `switch`-able companions of the string getters (which are unchanged): + +```cpp +auto obj = msg.object(0); +switch (obj.dtype_enum()) { // vs obj.dtype_string() + case tensogram::dtype::float32: /* … */ break; + default: break; +} +if (obj.byte_order_enum() == tensogram::byte_order::big) { /* … */ } +``` + +Both throw `tensogram::invalid_arg_error` for an out-of-range object index — the C accessors can only return their zero variant there, so the wrapper bounds-checks through the paired string getter first. + +`encode_options` gained two fields: + +```cpp +tensogram::encode_options opts; +opts.aggregate_hash = tensogram::aggregate_hash_policy::both; +opts.codec_backend = tensogram::compression_backend::pure; +auto bytes = tensogram::encode(meta_json, objects, opts); +``` + +- `aggregate_hash` — where the aggregate hash frame goes. Ignored when hashing is off; `::header` and `::both` are buffered-mode only and make `streaming_encoder` throw `tensogram::encoding_error`. +- `codec_backend` — which szip / zstd implementation to prefer. Spelled `codec_backend` because a member named `compression_backend` would shadow the enum type inside the struct. + +There are deliberately **no** `*_with_encode_options` overloads: they would be signature-identical to the existing ones. Instead the existing entry points — `encode()`, `file::append()`, `streaming_encoder` — pick the narrowest C function that can carry what you actually set, escalating to `*_with_encode_options` only when one of these two fields is non-default. A caller who sets neither keeps the exact previous call path (and the exact previous bytes). The one exception the C ABI imposes is `encode_pre_encoded()`, which has no full-option entry point and therefore **rejects** both fields rather than silently ignoring them. + ## Examples -See `examples/cpp/` for complete working examples covering encode/decode, metadata, file API, simple packing, and iterators. +See `examples/cpp/` for complete working examples covering encode/decode, metadata, file API, simple packing, and iterators. `examples/cpp/26_frame_walker.cpp` covers the frame walker, the message header, the typed enums, and the new `encode_options` fields. diff --git a/docs/src/guide/fortran-api.md b/docs/src/guide/fortran-api.md index 5397f246..6c003823 100644 --- a/docs/src/guide/fortran-api.md +++ b/docs/src/guide/fortran-api.md @@ -9,8 +9,9 @@ The library compiles clean under `-std=f2008 -Wall -Wextra -Werror`; see > **Status.** Emerging. The synchronous surface is complete: generic > encode/decode, the multi-message file API, application metadata, the -> encoding pipeline, and the streaming encoder. Only the async surface is -> deferred. +> encoding pipeline, the streaming encoder, the frame walker + message +> header, synchronous remote reads, and the typed dtype / byte-order / +> option enums. Only the async surface is deferred. ## The memory-order contract (read this first) @@ -353,6 +354,167 @@ object out with `tensogram_to_array`. See Planned next: the async surface (deferred). +## Frame walker and message header + +Two procedures describe a message's *structure* without decoding a +payload or a byte of CBOR — `tensogram_message_header_read` for the +24-byte envelope and `tensogram_frames` for a lazy walk over its frames. +Both take the bytes of **one** message (use `tensogram_scan` on a +multi-message buffer and pass each slice): + +```fortran +type(tensogram_message_header) :: hdr +type(tensogram_frame_iterator) :: it +type(tensogram_frame) :: fr +integer(c_int8_t), allocatable :: payload(:) +integer(c_int) :: err +logical :: found + +call tensogram_message_header_read(wire, hdr, err) +call tensogram_check(err, 'message_header_read') +print *, hdr%version(), hdr%total_length(), hdr%has_header_index() + +call tensogram_frames(wire, it, err) ! takes its own copy of `wire` +call tensogram_check(err, 'frames') +do + call it%next(fr, found, err) + if (.not. found) exit + payload = fr%payload() ! an independent copy + print *, fr%frame_type(), fr%offset(), fr%length(), size(payload), & + fr%has_hash() +end do +if (err /= TGM_ERROR_OK) print *, 'malformed: ', tensogram_last_error() +call it%free() +``` + +- `tensogram_frame` is a plain **value**: freely copyable, owns nothing to + release, and readable after both the iterator and the message buffer are + gone (its `%payload()` bytes were copied out). +- `tensogram_frame_iterator` is a **non-copyable handle** like every other + handle in this binding. It keeps a private copy of the message, so your + array may be modified or deallocated while the walk continues. Release + it with `%free()` (the finalizer does the same at scope exit). +- `%offset()` is a **1-based** index, matching `tensogram_scan`: the frame + occupies `wire(offset : offset + length - 1)`. `%payload()` is the + frame's *content*, with the 16-byte frame header and the type-specific + footer (20 bytes for `TGM_FRAME_TYPE_NTENSOR`, 12 otherwise) stripped. +- A `.false.` `found` ends the loop for both of the C ABI's stop + conditions; the optional `err` is the only place they differ — + `TGM_ERROR_OK` for a clean end, `TGM_ERROR_FRAMING` for a malformed + chain, `TGM_ERROR_INVALID_ARG` for a dead cursor (`%next` after + `%free`). +- The eight `hdr%has_*()` predicates name the optional frames. They are + exact for a buffered message; for a streaming one only + "frame present ⇒ flag set" holds, and `%total_length()` may be `0`. + +`TGM_FRAME_TYPE_HEADER_METADATA` … `TGM_FRAME_TYPE_NTENSOR` are exported +for `SELECT CASE` on `fr%frame_type()` (there is no parameter for the +reserved type 4). The full cross-language contract is in +[Frame Introspection](frame-introspection.md); see +[`examples/fortran/frame_walker.f90`](https://github.com/ecmwf/tensogram/blob/main/examples/fortran/frame_walker.f90) +for a runnable tour. + +## Reading a remote `.tgm` + +`tensogram_file_open_remote` opens an S3 / GCS / Azure / HTTP source +**synchronously** and hands back an ordinary `tensogram_file`, so the +whole file API above works unchanged and `file%close()` closes it as +usual: + +```fortran +type(tensogram_file) :: f +integer(c_int) :: err + +if (tensogram_is_remote_url(source)) then + ! No storage options: the short form. + call tensogram_file_open_remote(source, f, err) + ! With backend options — parallel key / value arrays: + call tensogram_file_open_remote(source, ['aws_region'], ['eu-west-1'], & + f, err, bidirectional = .true.) +else + call tensogram_file_open(source, f, err) +end if +call tensogram_check(err, 'open') +``` + +`tensogram_is_remote_url` recognises `s3`, `s3a`, `gs`, `az`, `azure`, +`http` and `https` (case-insensitively); plain paths and `file://` URLs +belong to `tensogram_file_open`. `bidirectional` (default `.true.`) +selects the meet-in-the-middle remote scan. + +`err` is `TGM_ERROR_INVALID_ARG` when `keys` and `values` have different +lengths (the C ABI takes one option count for both), and +`TGM_ERROR_REMOTE` for an unparseable URL, a missing object, a rejected +storage option, or a transport failure. + +> **The C library must be built with the opt-in `remote` Cargo feature** +> — it is **not** in the published C-API tarballs. Without it, +> `tensogram_is_remote_url` answers `.false.` for *every* input and +> `tensogram_file_open_remote` returns `TGM_ERROR_REMOTE` with a +> `tensogram_last_error()` message naming `--features=remote`. Argument +> validation runs before that check, so a genuine mistake is reported as +> a mistake in either build. See +> [C API — synchronous remote access](c-api.md#synchronous-remote-access-opt-in-remote-feature). + +## Typed enums and the full encode options + +The dtype / byte-order **string** getters are unchanged; each now has a +`SELECT CASE`-able companion returning a `TGM_DTYPE_*` / +`TGM_BYTE_ORDER_*` code: + +```fortran +integer(c_int) :: dt, err + +dt = tensogram_object_dtype_enum(msg, 1, err) ! `err` is optional +select case (dt) +case (TGM_DTYPE_FLOAT32) ; print *, '4 bytes per element' +case (TGM_DTYPE_FLOAT64) ; print *, '8 bytes per element' +end select +``` + +An enum result has no spare code for failure, so an out-of-range `iobj` +yields the **zero variant** (`TGM_DTYPE_FLOAT16` / +`TGM_BYTE_ORDER_LITTLE`) and sets the optional `err` to +`TGM_ERROR_INVALID_ARG` — pass `err` (or check +`tensogram_num_objects`) when the index is not already known good. + +Three `*_with_options` procedures expose the full encode-side option set, +including the two knobs that previously had no Fortran surface — where the +aggregate hash frame goes, and which codec backend to use. Every argument +after `err` is optional and defaults to the library default: + +```fortran +integer(c_int8_t), allocatable :: data(:) +integer(c_size_t) :: lens(1) +type(tensogram_buffer) :: buf +integer(c_int) :: err + +data = transfer(field, [0_c_int8_t], size(field) * 4) +lens(1) = int(size(field) * 4, c_size_t) +call tensogram_encode_with_options(descriptors_json, data, lens, buf, err, & + hash = 'xxh3', & + aggregate_hash = TGM_AGGREGATE_HASH_POLICY_BOTH, & + compression_backend = TGM_COMPRESSION_BACKEND_PURE) +``` + +| Procedure | Target | +|---|---| +| `tensogram_encode_with_options(metadata_json, data, lens, buf, err, …)` | a buffer | +| `tensogram_file_append_with_options(file, metadata_json, data, lens, err, …)` | an open file | +| `tensogram_streaming_encoder_create_with_options(path, enc, err, …)` | a streaming encoder | + +These take raw bytes rather than a typed array: `metadata_json` is the +full `{"descriptors":[…]}` envelope, `data` is every object's element +bytes **concatenated**, and `lens(k)` is object *k*'s byte length (so +`num_objects == size(lens)` and `sum(lens) == size(data)`). The generic +`tensogram_encode` / `tensogram_file_append` remain the convenient path +when you do not need these knobs. + +`TGM_AGGREGATE_HASH_POLICY_HEADER` and `..._BOTH` are **buffered-mode +only**: the streaming constructor rejects them with `TGM_ERROR_ENCODING`, +because a streaming writer emits its header before any data object exists. +Verify where the frame actually landed with the frame walker above. + ## Edge cases The binding fails gracefully rather than crashing: @@ -388,4 +550,6 @@ NumPy (when a Python with the `tensogram` package is present). ## See also - [C API](c-api.md) — the ABI the binding sits on. +- [Frame Introspection](frame-introspection.md) — the language-neutral + contract behind `tensogram_frames` / `tensogram_message_header_read`. - [Objects and Dtypes](../concepts/objects.md) — strides and layout. diff --git a/docs/src/guide/frame-introspection.md b/docs/src/guide/frame-introspection.md new file mode 100644 index 00000000..5ffed79a --- /dev/null +++ b/docs/src/guide/frame-introspection.md @@ -0,0 +1,376 @@ +# Frame Introspection + +Every tensogram message is a chain of **frames** wrapped in a fixed +envelope. This page is the single, language-neutral contract for reading +that structure — which frames a message holds, where each one starts, how +long it is, what its content bytes are, and whether its hash slot is +populated — **without decoding a payload or a byte of CBOR**. The same two +capabilities are available in **Rust, C, C++, Python, TypeScript, and +Fortran**, with matching semantics. + +## The model + +Two calls, each taking the bytes of **one** message: + +- **The message header** — the 24-byte preamble as typed values: the wire + `version`, the whole-message `total_length`, and eight `has_*` + predicates naming the optional frames. It reads 24 bytes and walks + nothing, so it is the cheapest way to tell a *random-access* message + (metadata / index / hashes in the **header**) from a *streaming* one + (in the **footer**). +- **The frame walk** — one record per frame, in wire order, carrying the + frame's type, its frame-header `version` and raw `flags`, its `offset` + and `length` within the message, its content bytes, and a `has_hash` + predicate. + +| Language | Frame walk | Message header | +|---|---|---| +| Rust | `frames(message) -> Result` | `message_header(message) -> Result` | +| C | `tgm_frame_iter_create` / `_next` / `_free` | `tgm_message_header` | +| C++ | `frames(msg, len) -> frame_range` | `read_message_header(msg, len)` | +| Python | `frames(buf) -> FrameIter` | `message_header(buf)` | +| TypeScript | `frames(buf) -> Frame[]` | `messageHeader(buf)` | +| Fortran | `tensogram_frames(buffer, iterator, err)` | `tensogram_message_header_read(buffer, header, err)` | + +### What counts as a frame + +The walk yields the message's frames and nothing else — types **1–3 and +5–9**: + +| Code | Name | Phase | Carries | +|---|---|---|---| +| 1 | `HeaderMetadata` | header | CBOR global metadata | +| 2 | `HeaderIndex` | header | CBOR index of data-object offsets | +| 3 | `HeaderHash` | header | CBOR aggregate of the per-object hashes | +| 5 | `FooterHash` | footer | CBOR aggregate of the per-object hashes | +| 6 | `FooterIndex` | footer | CBOR index of data-object offsets | +| 7 | `FooterMetadata` | footer | CBOR global metadata | +| 8 | `PrecederMetadata` | body | per-object metadata for the next data object | +| 9 | `NTensorFrame` | body | one data object: payload + masks + descriptor | + +These numbers are the wire's frame-type field, not a binding invention +(see [Message Layout](../format/wire-format.md#frame-types)). **Type 4 is +reserved** — it held the obsolete v2 data-object layout — which is why +the sequence skips from 3 to 5; a message that contains one is reported +as malformed. Only the spelling of the names differs per language: +`FrameType::NTensorFrame` (Rust), `TGM_FRAME_TYPE_NTENSOR` (C, Fortran), +`frame_type::ntensor` (C++), `"NTensorFrame"` (Python, TypeScript). + +> **The preamble and postamble are not frames.** They are the envelope, +> they are never yielded by the walk, and everything they hold is +> available from the message header instead. Inter-frame alignment +> padding is likewise part of no frame: the walk steps over it. + +### One message per call + +Both calls describe exactly **one** message and must be handed bytes that +start at its `TENSOGRM` preamble magic. A `.tgm` file — and any +multi-message buffer — is a plain concatenation of messages, so find the +boundaries with `scan()` first and slice: + +```text +scan(buf) → [(offset, length), …] one entry per message + └── frames(buf[offset .. offset + length]) +``` + +Frame offsets are then relative to **that slice**, not to the file; add +the message offset back when you want a file-absolute position. Passing +an unsliced multi-message buffer is not an error, but the answer only +describes the first message. + +### Offsets, spans, and payload boundaries + +- **`offset`** — byte position of the frame's 16-byte frame header, + relative to the start of the message that was passed in. It is + **0-based in every binding except Fortran**, which reports a 1-based + index (matching `tensogram_scan`), so the frame there occupies + `buffer(offset : offset + length - 1)`. +- **`length`** — the whole-frame span: frame header through the closing + `ENDF` marker, excluding any alignment padding that follows. +- **`payload`** — the frame's **content**, with the 16-byte frame header + *and* the type-specific footer stripped. The footer is **20 bytes** for + the data-object frame type (`[cbor_offset][hash][ENDF]`) and **12 + bytes** for every other type (`[hash][ENDF]`). + +```text + offset offset + length + │ │ + ▼ ▼ + ┌──────────────┬───────────────────┬──────────┐ + │ frame header │ payload (content) │ footer │ (padding) + │ 16 B │ │ 12 / 20 B│ + └──────────────┴───────────────────┴──────────┘ +``` + +So `payload` is `length - 16 - footer` bytes long. For a data-object +frame it is the encoded tensor payload, any NaN / Inf mask blobs, and the +trailing CBOR descriptor; for every other frame type it is the CBOR body. +Use `offset` / `length` when you want the whole frame — including its +`FR` header and `ENDF` marker — rather than just the content. + +### Lifetime: who owns the payload bytes + +| Binding | Payload | Contract | +|---|---|---| +| Rust | **borrowed** | `FrameInfo::payload` is a `&[u8]` view into the message slice; the compiler enforces the lifetime. | +| C | **borrowed** | `TgmFrame::payload` points *into* the `msg` buffer you passed to `tgm_frame_iter_create`. Never freed. It stays valid for as long as `msg` lives — later `_next` calls and `tgm_frame_iter_free` do **not** invalidate it. The cursor borrows `msg` for its whole lifetime, so `msg` must outlive the iterator and must not be moved, reallocated, or mutated meanwhile. | +| C++ | **borrowed** | `frame::payload()` / `payload_data()` view the same buffer, with the same rule: the buffer must outlive the `frame_range`, and the views survive the range's destruction. | +| Python | copied | `Frame.payload` materialises `bytes` on access from the source buffer the iterator retains, so the walk itself copies nothing and the frames stay valid independently of the caller's object. | +| TypeScript | copied | `Frame.payload` is a `Uint8Array` on the JS heap, not a view into WASM linear memory — safe to retain across later WASM calls and safe to mutate. | +| Fortran | copied | `frame%payload()` returns an independent `integer(c_int8_t)` array. `tensogram_frames` also keeps a **private copy of the message**, so the caller's array may be modified, deallocated, or go out of scope while the walk continues. | + +In the borrowing bindings, copy the bytes out if you need them to outlive +the message buffer. + +### Header flags: exact when buffered, advisory when streaming + +The eight predicates are the preamble's structural flags decoded into +named booleans: + +| Predicate | Meaning | +|---|---| +| `has_header_metadata` / `has_footer_metadata` | a metadata frame is present in the header / footer | +| `has_header_index` / `has_footer_index` | an object-index frame is present in the header / footer | +| `has_header_hashes` / `has_footer_hashes` | an aggregate-hash frame is present in the header / footer | +| `has_preceder_metadata` | at least one `PrecederMetadata` frame appears in the body | +| `has_hashes_present` | advisory: every frame has its per-frame `HASH_PRESENT` bit set | + +For a **buffered** message (one produced by `encode` / `append`) the +encoder knows the whole message before it writes the preamble, so every +flag is an exact statement about the frames present, and `total_length` +is the real byte count. + +For a **streaming** message the preamble is written before the first +object exists, so the flags are *advisory*. Only one direction is +guaranteed: + +> **frame present ⇒ flag set.** The converse does not hold. In +> particular the streaming encoder sets the `PRECEDER_METADATA` flag +> unconditionally — it cannot know at preamble-write time whether any +> preceder will follow — so `has_preceder_metadata` may be `true` for a +> message whose body holds no `PrecederMetadata` frame. Walk the frames +> if you need certainty. +> +> Note that the flag table in `plans/WIRE_FORMAT.md` §3.1 (and its +> rendering under [Message Layout](../format/wire-format.md#preamble-flags)) +> states the flag definitively; the behaviour described here is what the +> encoder actually writes. + +`total_length` is `0` in a streaming message whose length was never +back-filled (`finish()` rather than the back-filling variant). That is +not an error — it means "unknown at write time" — and the walk still +stops cleanly at the postamble. + +Two more asymmetries worth knowing: + +- A streaming message carries a `HeaderMetadata` frame *as well as* the + footer metadata / index, so `has_header_metadata` alone does not + identify a random-access layout. Use `has_header_index`. +- `has_hashes_present` is a coarse message-wide summary. For any single + frame the per-frame `has_hash` predicate is **authoritative**. + +### End of walk vs malformed chain + +A walk stops for two very different reasons, and every binding keeps them +distinguishable. Frames that parsed before the damage are always yielded +first — corruption is never silently rendered as "fewer frames". + +| Binding | Clean end | Malformed chain | Unreadable preamble | +|---|---|---|---| +| Rust | iterator returns `None` | one `Err` item, then the iterator stops | `frames()` / `message_header()` return `Err` | +| C | `tgm_frame_iter_next` returns `false` **and** `tgm_last_error()` is `NULL` | `_next` returns `false` with the reason in `tgm_last_error()`; iteration stays stopped | `tgm_frame_iter_create` returns `NULL`; `tgm_message_header` returns an error code | +| C++ | the range-for loop ends | `framing_error` thrown from the increment that finds it | `framing_error` from `frames()` / `read_message_header()` (`invalid_arg_error` for a null pointer) | +| Python | `StopIteration` | `ValueError` from `next()`, in position | `ValueError` from `frames()` / `message_header()` | +| TypeScript | the returned array ends | `FramingError` from `frames()` (no short array) | `FramingError`; `InvalidArgumentError` for a non-`Uint8Array` | +| Fortran | `found = .false.` with `err == TGM_ERROR_OK` | `found = .false.` with `err == TGM_ERROR_FRAMING`; `tensogram_last_error()` says why | `err` from `tensogram_frames` / `tensogram_message_header_read` | + +Laziness follows the host idiom: Rust, C, C++, and Fortran pull one frame +per step; Python performs the (header-only) structural walk up front and +materialises one `Frame` per `next()`; TypeScript returns the whole array, +because the underlying WASM call already materialises it. + +## By language + +All examples answer the same questions about one message: *is this a +random-access or a streaming layout?*, *which frames does it hold, and +where?*, *how many data objects are there?* + +### Rust + +```rust +use tensogram::{frames, message_header}; + +// message: &[u8] — ONE message (e.g. a slice located with `scan`) +// `version` / `total_length` / `flags` are plain fields on MessageHeader; +// the eight structural predicates are methods. +let header = message_header(message)?; +if header.has_header_index() { + // metadata + index are in the header: random access is cheap + println!("random-access, {} bytes", header.total_length); +} + +let mut data_objects = 0; +for frame in frames(message)? { + let f = frame?; // one Err, then the walk stops + if f.frame_type.is_data_object() { + data_objects += 1; + let _content = f.payload; // &[u8] borrowed from `message` + } + let _whole_frame = &message[f.offset..f.offset + f.length]; + let _hashed = f.has_hash(); // per-frame, authoritative +} +``` + +### C + +The cursor borrows `msg`; `msg` must outlive it. `TgmFrame::payload` +points into `msg` and is never freed. + +```c +TgmMessageHeader h; +if (tgm_message_header(msg, msg_len, &h) != TGM_ERROR_OK) { + fprintf(stderr, "not a message: %s\n", tgm_last_error()); + return 1; +} +printf("v%u, %llu bytes, header index=%d\n", (unsigned)h.version, + (unsigned long long)h.total_length, (int)h.has_header_index); + +tgm_frame_iter_t *it = tgm_frame_iter_create(msg, msg_len); +if (it == NULL) { /* tgm_last_error() says why */ return 1; } + +TgmFrame f; +size_t data_objects = 0; +while (tgm_frame_iter_next(it, &f)) { + if (f.frame_type == TGM_FRAME_TYPE_NTENSOR) data_objects++; + printf("type=%d offset=%zu length=%zu payload=%zu hash=%d\n", + (int)f.frame_type, f.offset, f.length, f.payload_len, + (int)tgm_frame_has_hash(&f)); +} +/* false means either a clean end or a broken chain: */ +const char *err = tgm_last_error(); /* NULL => clean end */ +tgm_frame_iter_free(it); /* payload pointers stay valid */ +``` + +### C++ + +```cpp +const auto h = tensogram::read_message_header(msg.data(), msg.size()); +if (h.has_footer_index()) { /* streaming layout */ } + +std::size_t data_objects = 0; +for (const auto& f : tensogram::frames(msg.data(), msg.size())) { + if (f.is_data_object()) ++data_objects; + std::string_view content = f.payload(); // borrows `msg` + (void)content; +} +// A broken chain throws tensogram::framing_error from the increment +// that discovers it, after the intact frames have been yielded. +``` + +`frame_range` is move-only and single-pass: it owns the C cursor (freed +in its destructor) and a second `begin()` resumes where the previous +iterator stopped. + +### Python + +```python +header = tensogram.message_header(msg) +header.has_header_index # True → random-access layout +header.total_length # 0 for a non-back-filled streaming message +header.flags # raw bits, if you really want them + +for frame in tensogram.frames(msg): + frame.frame_type # "NTensorFrame", "HeaderMetadata", … + frame.frame_type_code # 9, 1, … (the wire number) + frame.offset # start of the frame within `msg` + frame.length # whole-frame span, through ENDF + frame.payload # bytes; header + footer already stripped + frame.has_hash, frame.is_data_object + +# one message at a time +for offset, length in tensogram.scan(buf): + message = buf[offset:offset + length] + n = sum(1 for f in tensogram.frames(message) if f.is_data_object) +``` + +`FrameIter` also implements `len()` (frames left to yield) and `repr()`, +so it is comfortable at a REPL. + +### TypeScript + +```ts +import { frames, messageHeader, scan } from '@ecmwf.int/tensogram'; + +const header = messageHeader(msg); +header.hasHeaderIndex; // random-access layout +header.totalLength; // 0 for a non-back-filled streaming message + +for (const f of frames(msg)) { + f.frameType; // 'NTensorFrame' | 'HeaderMetadata' | … + f.frameTypeCode; // 9 | 1 | … + f.offset; // start of the frame within `msg` + f.length; // whole-frame span, through `ENDF` + f.payload; // Uint8Array copy on the JS heap + f.hasHash; +} + +// one message at a time +for (const { offset, length } of scan(fileBytes)) { + const chain = frames(fileBytes.subarray(offset, offset + length)); + const objects = chain.filter((f) => f.frameType === 'NTensorFrame'); + console.log(offset, chain.length, objects.length); +} +``` + +### Fortran + +Offsets are **1-based**; the iterator is a non-copyable handle that keeps +its own copy of the message, and every payload is copied out. + +```fortran +type(tensogram_message_header) :: hdr +type(tensogram_frame_iterator) :: it +type(tensogram_frame) :: fr +integer(c_int8_t), allocatable :: payload(:) +integer(c_int) :: err +integer :: data_objects +logical :: found + +call tensogram_message_header_read(wire, hdr, err) +call tensogram_check(err, 'message_header_read') +print *, hdr%version(), hdr%total_length(), hdr%has_header_index() + +call tensogram_frames(wire, it, err) ! takes its own copy of `wire` +call tensogram_check(err, 'frames') +data_objects = 0 +do + call it%next(fr, found, err) + if (.not. found) exit + if (fr%frame_type() == TGM_FRAME_TYPE_NTENSOR) data_objects = data_objects + 1 + payload = fr%payload() ! an independent copy + print *, fr%frame_type(), fr%offset(), fr%length(), size(payload), & + fr%has_hash() +end do +if (err /= TGM_ERROR_OK) print *, 'malformed chain: ', tensogram_last_error() +call it%free() +``` + +## Runnable examples + +| Language | Example | +|---|---| +| C++ | [`examples/cpp/26_frame_walker.cpp`](https://github.com/ecmwf/tensogram/blob/main/examples/cpp/26_frame_walker.cpp) | +| Python | [`examples/python/21_frames_and_message_header.py`](https://github.com/ecmwf/tensogram/blob/main/examples/python/21_frames_and_message_header.py) | +| TypeScript | [`examples/typescript/21_frame_walker.ts`](https://github.com/ecmwf/tensogram/blob/main/examples/typescript/21_frame_walker.ts) | +| Fortran | [`examples/fortran/frame_walker.f90`](https://github.com/ecmwf/tensogram/blob/main/examples/fortran/frame_walker.f90) | + +## See also + +- [What is a Message?](../concepts/messages.md) — why the format is + frame-based in the first place. +- [Message Layout](../format/wire-format.md) — the byte-level spec for + the preamble, frame header, footers, and every frame type. +- [Reading Metadata](metadata.md) — the companion contract for reading a + frame's *contents* rather than its structure. +- [Iterators](iterators.md) — walking *messages* and *objects*, one level + up from frames. diff --git a/docs/src/guide/python-api.md b/docs/src/guide/python-api.md index b1823cf2..333c3031 100644 --- a/docs/src/guide/python-api.md +++ b/docs/src/guide/python-api.md @@ -151,6 +151,26 @@ msg = tensogram.encode( ) ``` +### Message-level encode options + +Two keyword arguments steer the writer rather than a single object's pipeline. Both are accepted by `encode()`, `TensogramFile.append()`, `StreamingEncoder(...)`, and `AsyncStreamingEncoder.create(...)`: + +```python +msg = tensogram.encode( + {}, + [(desc, data)], + aggregate_hash="both", # "auto" (default) | "none" | "header" | "footer" | "both" + compression_backend="pure", # "auto" (default) | "ffi" | "pure" +) +``` + +| Argument | What it does | +|-----------|--------------| +| `aggregate_hash` | Where the aggregate hash frame (the redundant list of every per-object digest) is written. `"auto"` resolves to the header when buffering and the footer when streaming; `"none"` omits the frame entirely. Ignored when `hash=None` — there is nothing to aggregate. | +| `compression_backend` | Which implementation to use for `szip` / `zstd` when both a C and a pure-Rust one are compiled in. `"auto"` consults `TENSOGRAM_COMPRESSION_BACKEND`, then the platform default. Purely an implementation choice: the decoded values are identical either way. | + +An unknown value for either raises `ValueError`. On a `StreamingEncoder`, `aggregate_hash="header"` / `"both"` also raise `ValueError`: the streaming header is written before any data object exists, so the per-object hashes are not yet known. Use [`frames()`](frame-introspection.md) to see where the frame actually landed. + ### Pre-encoded data If you already have compressed/packed payloads (e.g. from another system), use `tensogram.encode_pre_encoded()` with the same interface. The library skips the encoding pipeline and writes the bytes as-is: @@ -240,6 +260,28 @@ for meta, objects in tensogram.iter_messages(buf): print(meta.version, len(objects)) ``` +### Frame introspection + +`tensogram.message_header()` and `tensogram.frames()` describe a message's *structure* without decoding a payload or a byte of CBOR. Both take exactly **one** message, so slice a multi-message buffer with `scan()` first: + +```python +header = tensogram.message_header(msg) +header.version # 3 +header.total_length # 0 for a non-back-filled streaming message +header.has_header_index # True → random-access layout + +for frame in tensogram.frames(msg): + print(frame.frame_type, # "HeaderMetadata", "NTensorFrame", … + frame.frame_type_code, # the numeric wire code + frame.offset, # start of the frame within `msg` + frame.length, # whole-frame span: header through ENDF + len(frame.payload), # content only: header + footer stripped + frame.has_hash, + frame.is_data_object) +``` + +`MessageHeader` also exposes `flags` plus the other seven `has_*` predicates (`has_footer_metadata`, `has_header_hashes`, …). `FrameIter` supports `len()` (frames left to yield) and a readable `repr()`; `Frame.payload` is `bytes`, materialised on access from the buffer the iterator retains, so walking a large message purely for offsets copies nothing. A malformed frame chain raises `ValueError` from `next()` *after* the frames that did parse; an unreadable preamble raises from `frames()` itself. See [Frame Introspection](frame-introspection.md) for the full contract and `examples/python/21_frames_and_message_header.py` for a runnable tour. + ### Hash verification ```python diff --git a/docs/src/guide/typescript-api.md b/docs/src/guide/typescript-api.md index 800e96a3..c921e915 100644 --- a/docs/src/guide/typescript-api.md +++ b/docs/src/guide/typescript-api.md @@ -87,9 +87,27 @@ await init({ wasmInput: new URL('...', import.meta.url) }); // custom location | `metadata` | `GlobalMetadata` | Free-form metadata; only `base`, `_reserved_`, `_extra_` are library-interpreted. An empty `{}` is valid. The wire-format version lives in the preamble — see [`WIRE_VERSION`](#wire_version). | | `objects` | `Array<{ descriptor, data }>` | Each `data` is a `TypedArray` or `Uint8Array` | | `opts.hash` | `'xxh3' \| false` | Hash algorithm. Default `'xxh3'`. Pass `false` to disable. | +| `opts.aggregateHash` | `AggregateHashPolicy` | Where the aggregate hash frame goes: `'auto'` (default; this buffered encoder writes a `HeaderHash` frame), `'none'`, `'header'`, `'footer'`, `'both'`. Ignored when `hash: false`. | +| `opts.compressionBackend` | `CompressionBackend` | `'auto'` (default), `'ffi'`, `'pure'`. **No-op on WASM** — see below. | Returns: `Uint8Array` containing the complete wire-format message. +`aggregateHash` decides where the redundant list of per-object digests is +written; the per-frame inline hash slots stay authoritative either way. +Use [`frames()`](#framesbuf) to observe the resulting placement. + +`compressionBackend` is accepted and validated (an unrecognised name +throws `InvalidArgumentError`) but never changes behaviour here: this +bundle is compiled with the pure-Rust codecs only, because there is no C +library to link against in a browser or in Node's WASM sandbox. The knob +exists for source symmetry with the Rust, Python, and C bindings — code +written against one binding reads the same against another. The same +accepted asymmetry applies to the core's `threads` knob, which the +single-threaded WASM build does not expose at all. + +Both options are also accepted by `TensogramFile#append`. +`StreamingEncoder` and `encodePreEncoded` do not take them. + ### `decode(buf, opts?)` | Parameter | Type | Description | @@ -113,6 +131,59 @@ Returns `Array<{ offset: number; length: number }>` for each Tensogram message found in a (potentially multi-message) buffer. Garbage between messages is silently skipped. +### `frames(buf)` + +Returns `Frame[]` — one entry per frame of **one** message, in wire +order, without decoding any payload or CBOR. The preamble and postamble +are not frames and never appear. + +```ts +for (const f of frames(msg)) { + f.frameType; // 'HeaderMetadata' | 'HeaderIndex' | … | 'NTensorFrame' + f.frameTypeCode; // the numeric wire code (1–3, 5–9) + f.version; + f.flags; // bit 1 is HASH_PRESENT, surfaced as f.hasHash + f.offset; // start of the frame, relative to `buf` + f.length; // whole-frame span: frame header through `ENDF` + f.payload; // content only: frame header + footer stripped + f.hasHash; +} +``` + +The array is materialised eagerly — a message holds only a handful of +frames, and the underlying WASM call already builds the whole walk, so a +lazy generator would promise a laziness this binding cannot deliver. The +consequence is that a malformed frame chain **throws** `FramingError` +rather than returning a short array; corruption is never silently +rendered as "fewer frames". Each `payload` is a copy on the JS heap +(never a view into WASM linear memory), so it stays valid across later +WASM calls and can be mutated freely. + +For a multi-message buffer, slice with `scan()` first — frame offsets are +relative to the message you pass in: + +```ts +for (const { offset, length } of scan(fileBytes)) { + const chain = frames(fileBytes.subarray(offset, offset + length)); +} +``` + +### `messageHeader(buf)` + +Returns the `MessageHeader` of **one** message: `version`, +`totalLength`, and eight `has*` predicates (`hasHeaderMetadata`, +`hasFooterMetadata`, `hasHeaderIndex`, `hasFooterIndex`, +`hasHeaderHashes`, `hasFooterHashes`, `hasPrecederMetadata`, +`hasHashesPresent`). It parses 24 bytes and walks nothing, so it is the +cheapest way to tell a random-access message (metadata / index / hashes +in the *header*) from a streaming one (in the *footer*). + +The predicates are exact for a message produced by `encode()`. For one +produced by `StreamingEncoder` they are advisory — only +`frame present ⇒ flag set` holds, and `totalLength` may stay `0` — see +[Frame Introspection](frame-introspection.md) for the full contract and +`examples/typescript/21_frame_walker.ts` for a runnable tour. + ### `DecodedObject` / `DecodedFrame` ```ts @@ -671,6 +742,7 @@ See `examples/typescript/` in the repository for runnable scripts: - `12_streaming_encoder.ts` — frame-at-a-time encoder with preceders - `13_range_access.ts` — lazy `TensogramFile.fromUrl` over HTTP Range - `14_streaming_callback.ts` — `StreamingEncoder` with `onBytes` callback sink +- `21_frame_walker.ts` — `frames()` / `messageHeader()` structural introspection Run them with: diff --git a/examples/cpp/26_frame_walker.cpp b/examples/cpp/26_frame_walker.cpp new file mode 100644 index 00000000..5d39db00 --- /dev/null +++ b/examples/cpp/26_frame_walker.cpp @@ -0,0 +1,218 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +/// @file 26_frame_walker.cpp +/// @brief Example 26 — walk a message's frames, read its envelope, and steer +/// the encoder with the full `encode_options` set (C++ wrapper). +/// +/// Demonstrates: +/// * `tensogram::read_message_header()` — the 24-byte preamble as typed +/// predicates: is this message random-access (metadata / index / hashes +/// in the header) or streaming (in the footer)? +/// * `tensogram::frames()` — a lazy range over the frames of one message, +/// usable in a range-for. Each frame borrows the caller's buffer, so +/// the payload view stays valid after the range is gone. +/// * `tensogram::encode_options::aggregate_hash` — where the aggregate +/// hash frame goes, verified by walking the frames it produced; and +/// `codec_backend`, which picks the codec implementation. +/// * `tensogram::is_remote_url()` — whether a source belongs to the remote +/// backend (`file::open_remote()`) or the local one (`file::open()`). +/// +/// Build: +/// cmake -S cpp -B build -DCMAKE_BUILD_TYPE=Release +/// cmake --build build -j +/// ./build/bin/26_frame_walker + +#include + +#include +#include +#include +#include +#include + +namespace { + +/// A 1-D float32 descriptor, repeated @p n times (one per data object). +std::string f32_json(std::size_t n, std::size_t elements) { + std::string json = R"({"descriptors":[)"; + for (std::size_t i = 0; i < n; ++i) { + if (i != 0) json += ","; + json += R"({"type":"ndarray","ndim":1,"shape":[)" + std::to_string(elements) + + R"(],"strides":[4],"dtype":"float32","byte_order":"little",)" + R"("encoding":"none","filter":"none","compression":"none"})"; + } + return json + "]}"; +} + +const char* frame_name(tensogram::frame_type t) { + switch (t) { + case tensogram::frame_type::header_metadata: return "HeaderMetadata"; + case tensogram::frame_type::header_index: return "HeaderIndex"; + case tensogram::frame_type::header_hash: return "HeaderHash"; + case tensogram::frame_type::footer_hash: return "FooterHash"; + case tensogram::frame_type::footer_index: return "FooterIndex"; + case tensogram::frame_type::footer_metadata: return "FooterMetadata"; + case tensogram::frame_type::preceder_metadata: return "PrecederMetadata"; + case tensogram::frame_type::ntensor: return "NTensor"; + } + return "unknown"; +} + +/// Render the eight structural predicates as "header / footer / both / -". +std::string placement(bool in_header, bool in_footer) { + if (in_header && in_footer) return "both"; + if (in_header) return "header"; + if (in_footer) return "footer"; + return "-"; +} + +void print_header(const char* label, const std::vector& msg) { + const auto h = tensogram::read_message_header(msg.data(), msg.size()); + std::printf("%s: version=%u total_length=%llu%s\n", label, h.version(), + static_cast(h.total_length()), + h.total_length() == 0 ? " (streaming writer never back-filled it)" : ""); + std::printf(" metadata: %-6s index: %-6s aggregate hashes: %s\n", + placement(h.has_header_metadata(), h.has_footer_metadata()).c_str(), + placement(h.has_header_index(), h.has_footer_index()).c_str(), + placement(h.has_header_hashes(), h.has_footer_hashes()).c_str()); + std::printf(" preceder metadata: %s every frame hashed: %s\n", + h.has_preceder_metadata() ? "yes (advisory when streaming)" : "no", + h.has_hashes_present() ? "yes" : "no"); +} + +void print_frames(const std::vector& msg) { + // Lazy: one C-level `next` per loop iteration, nothing materialised up + // front. A malformed frame chain throws framing_error here; a clean end + // just terminates the loop. + for (const auto& f : tensogram::frames(msg.data(), msg.size())) { + std::printf(" %-16s offset=%-5zu length=%-5zu payload=%-5zu hash=%s%s\n", + frame_name(f.type()), f.offset(), f.length(), f.payload_size(), + f.has_hash() ? "yes" : "no ", + f.is_data_object() ? " <- data object" : ""); + } +} + +} // namespace + +int main() { + std::vector values{1.0f, 2.0f, 3.0f, 4.0f}; + std::vector> objects{ + {reinterpret_cast(values.data()), + values.size() * sizeof(float)}, + {reinterpret_cast(values.data()), + values.size() * sizeof(float)}}; + const std::string json = f32_json(objects.size(), values.size()); + + // ── 1. A buffered (random-access) message ───────────────────────────── + std::printf("=== buffered message ===\n"); + auto buffered = tensogram::encode(json, objects); + print_header("preamble", buffered); + print_frames(buffered); + + // ── 2. The payload borrows the caller's buffer ──────────────────────── + // frame::payload() points INTO `buffered`; it stays valid after the range + // (and its C cursor) are destroyed, for as long as `buffered` lives. + std::printf("\n=== borrowed payloads ===\n"); + std::vector collected; + { + auto range = tensogram::frames(buffered.data(), buffered.size()); + for (const auto& f : range) collected.push_back(f); + } // the cursor is freed here — the payload views are not + for (const auto& f : collected) { + const auto payload = f.payload(); + std::printf(" %-16s payload[0..4] =", frame_name(f.type())); + for (std::size_t i = 0; i < 4 && i < payload.size(); ++i) { + std::printf(" %02x", static_cast( + static_cast(payload[i]))); + } + std::printf(" (offset %zu into the message buffer)\n", + static_cast(f.payload_data() - buffered.data())); + } + + // ── 3. Steering the encoder: aggregate hash placement ───────────────── + // A reader that scans backwards wants the hash list next to the + // postamble; a reader that streams forwards wants it in the header. + // `both` writes identical lists in both places — verified by walking. + std::printf("\n=== encode_options: aggregate_hash = both ===\n"); + tensogram::encode_options opts; + opts.aggregate_hash = tensogram::aggregate_hash_policy::both; + opts.codec_backend = tensogram::compression_backend::automatic; + auto both = tensogram::encode(json, objects, opts); + print_header("preamble", both); + print_frames(both); + + // ── 4. A streaming message puts metadata and index in the footer ────── + std::printf("\n=== streaming message ===\n"); + const std::string path = "tensogram_example_26_stream.tgm"; + { + // header / both are rejected when streaming: the header is written + // before any data object exists, so the hashes are not yet known. + tensogram::encode_options stream_opts; + stream_opts.aggregate_hash = tensogram::aggregate_hash_policy::footer; + tensogram::streaming_encoder enc(path, "{}", stream_opts); + enc.write_preceder(R"({"units":"K"})"); + enc.write_object( + R"({"type":"ndarray","ndim":1,"shape":[4],"strides":[4],"dtype":"float32",)" + R"("byte_order":"little","encoding":"none","filter":"none","compression":"none"})", + reinterpret_cast(values.data()), + values.size() * sizeof(float)); + enc.finish(); + } + auto streamed = tensogram::file::open(path).read_message(0); + print_header("preamble", streamed); + print_frames(streamed); + std::remove(path.c_str()); + + // ── 5. Typed descriptor enums ───────────────────────────────────────── + std::printf("\n=== typed descriptor enums ===\n"); + auto decoded = tensogram::decode(buffered.data(), buffered.size()); + const auto obj = decoded.object(0); + std::printf(" dtype : %-10s (enum code %d)\n", obj.dtype_string().c_str(), + static_cast(obj.dtype_enum())); + std::printf(" byte order : %-10s (enum code %d)\n", obj.byte_order_string().c_str(), + static_cast(obj.byte_order_enum())); + switch (obj.dtype_enum()) { + case tensogram::dtype::float32: + std::printf(" switched on dtype::float32 — 4 bytes per element\n"); + break; + default: + std::printf(" some other dtype\n"); + break; + } + + // ── 6. Local or remote? ─────────────────────────────────────────────── + std::printf("\n=== is_remote_url ===\n"); + for (const char* source : {"s3://bucket/forecast.tgm", "https://host/forecast.tgm", + "/data/forecast.tgm", "file:///data/forecast.tgm"}) { + const bool remote = tensogram::is_remote_url(source); + std::printf(" %-30s -> %s\n", source, + remote ? "file::open_remote()" : "file::open()"); + } + std::printf("(a build without the `remote` Cargo feature answers \"local\" for\n" + " every URL — it genuinely cannot open one; rebuild the C API with\n" + " --features remote, or CMake -DTENSOGRAM_REMOTE=ON)\n"); + + // ── 7. A malformed frame chain is not a silent short walk ───────────── + std::printf("\n=== malformed frame chain ===\n"); + std::vector truncated(buffered.begin(), + buffered.begin() + + static_cast(buffered.size() - 64)); + try { + std::size_t seen = 0; + for (const auto& f : tensogram::frames(truncated.data(), truncated.size())) { + (void)f; + ++seen; + } + std::printf(" walked %zu frames with no error\n", seen); + } catch (const tensogram::framing_error& e) { + std::printf(" framing_error after the intact frames: %s\n", e.what()); + } + + return 0; +} diff --git a/examples/cpp/CMakeLists.txt b/examples/cpp/CMakeLists.txt index 2a4f4c11..ea1528de 100644 --- a/examples/cpp/CMakeLists.txt +++ b/examples/cpp/CMakeLists.txt @@ -15,6 +15,7 @@ set(EXAMPLES 13_validate 16_multi_threaded_pipeline 17_masks_scan_validate + 26_frame_walker ) # Async examples on the callback / std::future frontends (C++17, header-only). diff --git a/examples/cpp/README.md b/examples/cpp/README.md index 7f6fffdd..0b3b7a1c 100644 --- a/examples/cpp/README.md +++ b/examples/cpp/README.md @@ -29,6 +29,7 @@ C++17 interface over the C FFI. | `23_async_stdfuture.cpp` | Async read via the `std::future` frontend (C++17) | | `24_async_cancellation.cpp` | Cancellation tokens and timeouts (callback frontend) | | `25_async_decode_object_range.cpp` | Async single-object + partial-range decode, `path()`, and the pull-model task handle (`ready()`/`cancel()`/`join()`) | +| `26_frame_walker.cpp` | Lazy `frames()` walk + `read_message_header()` envelope, borrowed frame payloads, `encode_options::aggregate_hash` / `codec_backend`, typed `dtype_enum()`, and `is_remote_url()` | ### Async examples (19–25) @@ -48,6 +49,24 @@ See the [C++ Async API](../../docs/src/guide/cpp-async.md) and [C++ Async Streaming](../../docs/src/guide/cpp-streaming-async.md) guides for the API reference and the producer/consumer recipe. +### Synchronous remote sources + +`tensogram::is_remote_url()` and `tensogram::file::open_remote()` link in +every build, but they need the C API's `remote` Cargo feature to actually +reach an object store. Without it `is_remote_url()` answers `false` for +every URL (this build genuinely cannot open one) and `open_remote()` +throws `tensogram::remote_error` explaining how to enable it: + +```bash +cmake -S cpp -B build -DCMAKE_BUILD_TYPE=Release -DTENSOGRAM_REMOTE=ON +cmake --build build -j +``` + +`open_remote()` returns an ordinary `tensogram::file`, so `message_count()`, +`read_message()`, `decode_message()` and `file_iterator` work unchanged +against `s3://`, `gs://`, `az://`, `http(s)://` and `file://` sources. It is +independent of the async options — no async runtime is pulled in. + ## API Overview ```cpp @@ -66,9 +85,26 @@ msg.version(); // wire-format version msg.num_objects(); // number of data objects auto obj = msg.object(0); obj.dtype_string(); // "float32", "int64", etc. +obj.dtype_enum(); // tensogram::dtype::float32 — switchable +obj.byte_order_enum(); // tensogram::byte_order::little obj.shape(); // std::vector obj.data_as(); // typed pointer to payload +// Message envelope + lazy frame walk (payloads borrow `buf`) +auto header = tensogram::read_message_header(buf, len); +header.total_length(); // 0 for a streaming message never back-filled +for (const auto& f : tensogram::frames(buf, len)) { + f.type(); // tensogram::frame_type::ntensor, ... + f.offset(); // where the frame starts in `buf` + f.payload(); // std::string_view INTO `buf` + f.has_hash(); // per-frame HASH_PRESENT flag +} + +// Remote sources (needs the C API's `remote` feature) +tensogram::is_remote_url("s3://bucket/key.tgm"); +auto remote = tensogram::file::open_remote("s3://bucket/key.tgm", + {{"aws_region", "eu-west-1"}}); + // Range-based for over objects for (const auto& obj : msg) { ... } @@ -93,6 +129,12 @@ tensogram::streaming_encoder enc(path, metadata_json, opts); enc.write_object(descriptor_json, data, len); enc.finish(); +// Encode options (every field defaults to the library default) +tensogram::encode_options opts; +opts.hash_algo = "xxh3"; +opts.aggregate_hash = tensogram::aggregate_hash_policy::both; // buffered only +opts.codec_backend = tensogram::compression_backend::pure; // szip / zstd + // Utilities auto entries = tensogram::scan(buf, len); auto hash = tensogram::compute_hash(data, len, "xxh3"); diff --git a/examples/fortran/README.md b/examples/fortran/README.md index d9a69982..26b13b99 100644 --- a/examples/fortran/README.md +++ b/examples/fortran/README.md @@ -11,6 +11,7 @@ by `cargo cinstall -p tensogram-ffi` or the release tarballs). | `streaming.f90` | Stream a multi-object message progressively (one object at a time), then reopen and decode every object | | `validate.f90` | Scan a multi-message buffer for boundaries, read each message's wire version, and validate the buffer + a `.tgm` file (JSON reports) | | `masks.f90` | Wave-C reader/writer helpers: mask-aware decode (`decode_with_masks` + `object_mask`), `validate_buffer` + inline hashes, `scan_file` / `scan_with_options`, `compute_common`, and `verify_canonical_cbor` | +| `frame_walker.f90` | Look inside a message without decoding it: the typed message header (`message_header_read`), a lazy walk over every frame (`tensogram_frames` + `%next`, with copied payload bytes), and `encode_with_options(aggregate_hash = BOTH)` verified with the walker itself | ## Build & run diff --git a/examples/fortran/frame_walker.f90 b/examples/fortran/frame_walker.f90 new file mode 100644 index 00000000..3c126f79 --- /dev/null +++ b/examples/fortran/frame_walker.f90 @@ -0,0 +1,224 @@ +! (C) Copyright 2026- ECMWF and individual contributors. +! +! This software is licensed under the terms of the Apache Licence Version 2.0 +! which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +! In applying this licence, ECMWF does not waive the privileges and immunities +! granted to it by virtue of its status as an intergovernmental organisation nor +! does it submit to any jurisdiction. + +!> examples/fortran/frame_walker.f90 +!> +!> Look INSIDE a message without decoding it — the structural view the other +!> readers never expose: +!> * tensogram_message_header_read — the 24-byte preamble as a typed record: +!> wire version, total length, and eight named flags that say whether the +!> message is random-access (metadata / index / hashes in the HEADER) or +!> streaming (in the FOOTER). +!> * tensogram_frames + %next — a lazy walk over every frame, one +!> tensogram_frame value at a time: type, version, flags, offset, length, +!> inline-hash flag and the payload bytes (COPIED out of the message, so +!> they outlive both the iterator and the buffer). +!> * tensogram_encode_with_options — aggregate_hash = BOTH, whose effect is +!> then VERIFIED with the walker: a hash frame in the header AND one in +!> the footer. +!> +!> Mirrors the C++ `examples/cpp/26_frame_walker.cpp` and the Python +!> `examples/python/21_frames_and_message_header.py`. +program frame_walker + use, intrinsic :: iso_c_binding + use tensogram + implicit none + + character(len=*), parameter :: path = 'frame_walker_stream.tgm' + integer, parameter :: NI = 4, NJ = 3 + real(c_float) :: field(NI, NJ) + integer(c_int8_t), allocatable :: wire(:), data(:) + integer(c_size_t) :: lens(1) + type(tensogram_buffer) :: buf + integer(c_int) :: err + integer :: i, j + + do j = 1, NJ; do i = 1, NI; field(i, j) = real(i * 10 + j, c_float); end do; end do + + ! ── 1. A buffered (random-access) message ─────────────────────────────── + print '(a)', '=== buffered message ===' + call tensogram_encode(field, buf, err) + call tensogram_check(err, 'encode') + call buf%as_array(wire) + call show_header(wire) + call show_frames(wire) + + ! ── 2. A streaming message: metadata / index / hash move to the footer ── + print '(a)', '' + print '(a)', '=== streaming message ===' + call write_stream(path, field) + call read_first_message(path, wire) + call show_header(wire) + call show_frames(wire) + call delete_file(path) + + ! ── 3. aggregate_hash = BOTH, verified with the walker ────────────────── + print '(a)', '' + print '(a)', '=== encode_with_options(aggregate_hash = BOTH) ===' + data = transfer(field, [0_c_int8_t], NI * NJ * 4) + lens(1) = int(NI * NJ * 4, c_size_t) + call tensogram_encode_with_options(descriptors_json(NI * NJ), data, lens, buf, err, & + aggregate_hash=TGM_AGGREGATE_HASH_POLICY_BOTH) + call tensogram_check(err, 'encode_with_options') + call buf%as_array(wire) + print '(a,l1,a,l1)', 'header hash frame: ', has_frame(wire, TGM_FRAME_TYPE_HEADER_HASH), & + ' footer hash frame: ', has_frame(wire, TGM_FRAME_TYPE_FOOTER_HASH) + call show_frames(wire) + +contains + + !> Print the message envelope: version, byte count and the eight flags. + subroutine show_header(msg) + integer(c_int8_t), intent(in) :: msg(:) + type(tensogram_message_header) :: hdr + integer(c_int) :: err + call tensogram_message_header_read(msg, hdr, err) + call tensogram_check(err, 'message_header_read') + print '(a,i0,a,i0,a,i0,a)', 'preamble: version=', hdr%version(), & + ' total_length=', hdr%total_length(), ' (buffer holds ', size(msg), ' bytes)' + print '(a,l1,a,l1,a,l1,a,l1)', & + ' metadata: header=', hdr%has_header_metadata(), & + ' footer=', hdr%has_footer_metadata(), & + ' index: header=', hdr%has_header_index(), & + ' footer=', hdr%has_footer_index() + print '(a,l1,a,l1,a,l1,a,l1)', & + ' hashes: header=', hdr%has_header_hashes(), & + ' footer=', hdr%has_footer_hashes(), & + ' per-frame=', hdr%has_hashes_present(), & + ' preceder metadata=', hdr%has_preceder_metadata() + end subroutine show_header + + !> Walk every frame and print one line each. A .false. `found` ends the + !> loop; `err` then tells a clean end from a malformed frame chain. + subroutine show_frames(msg) + integer(c_int8_t), intent(in) :: msg(:) + type(tensogram_frame_iterator) :: it + type(tensogram_frame) :: fr + integer(c_int8_t), allocatable :: payload(:) + integer(c_int) :: err + integer :: n + logical :: found + call tensogram_frames(msg, it, err) + call tensogram_check(err, 'frames') + n = 0 + do + call it%next(fr, found, err) + if (.not. found) exit + n = n + 1 + payload = fr%payload() ! an independent copy of the bytes + print '(a,i0,a,a,a,i0,a,i0,a,i0,a,l1)', & + ' frame ', n, ': ', frame_name(fr%frame_type()), & + ' offset=', fr%offset(), ' length=', fr%length(), & + ' payload=', size(payload), ' hashed=', fr%has_hash() + end do + if (err /= TGM_ERROR_OK) then + ! Malformed chain: the frames before the damage were still yielded. + print '(a,a)', ' !! malformed frame chain: ', tensogram_last_error() + else + print '(a,i0,a)', ' (', n, ' frames, walk ended cleanly)' + end if + call it%free() + end subroutine show_frames + + !> .true. when the message contains at least one frame of type `ftype`. + logical function has_frame(msg, ftype) + integer(c_int8_t), intent(in) :: msg(:) + integer(c_int), intent(in) :: ftype + type(tensogram_frame_iterator) :: it + type(tensogram_frame) :: fr + integer(c_int) :: err + logical :: found + has_frame = .false. + call tensogram_frames(msg, it, err) + call tensogram_check(err, 'frames') + do + call it%next(fr, found, err) + if (.not. found) exit + if (fr%frame_type() == ftype) has_frame = .true. + end do + call it%free() + end function has_frame + + !> Human-readable label for a TGM_FRAME_TYPE_* code (4 is reserved). + function frame_name(ftype) result(name) + integer(c_int), intent(in) :: ftype + character(len=17) :: name + select case (ftype) + case (TGM_FRAME_TYPE_HEADER_METADATA); name = 'header metadata ' + case (TGM_FRAME_TYPE_HEADER_INDEX); name = 'header index ' + case (TGM_FRAME_TYPE_HEADER_HASH); name = 'header hash ' + case (TGM_FRAME_TYPE_FOOTER_HASH); name = 'footer hash ' + case (TGM_FRAME_TYPE_FOOTER_INDEX); name = 'footer index ' + case (TGM_FRAME_TYPE_FOOTER_METADATA); name = 'footer metadata ' + case (TGM_FRAME_TYPE_PRECEDER_METADATA); name = 'preceder metadata' + case (TGM_FRAME_TYPE_NTENSOR); name = 'ntensor (data) ' + case default; name = 'unknown ' + end select + end function frame_name + + !> Stream one object to `path` (metadata / index / hash land in the footer). + subroutine write_stream(file_path, a) + character(len=*), intent(in) :: file_path + real(c_float), intent(in) :: a(:,:) + type(tensogram_streaming_encoder) :: enc + integer(c_int) :: err + call tensogram_streaming_encoder_create(file_path, enc, err) + call tensogram_check(err, 'streaming create') + call tensogram_streaming_encoder_write(enc, a, err) + call tensogram_check(err, 'streaming write') + call tensogram_streaming_encoder_finish(enc, err) + call tensogram_check(err, 'streaming finish') + call enc%free() + end subroutine write_stream + + !> Read message 1 of `path` back as raw wire bytes. + subroutine read_first_message(file_path, msg) + character(len=*), intent(in) :: file_path + integer(c_int8_t), allocatable, intent(out) :: msg(:) + type(tensogram_file) :: f + type(tensogram_buffer) :: raw + integer(c_int) :: err + call tensogram_file_open(file_path, f, err) + call tensogram_check(err, 'file_open') + call tensogram_file_read_message(f, 1, raw, err) + call tensogram_check(err, 'read_message') + call raw%as_array(msg) + call f%close() + end subroutine read_first_message + + subroutine delete_file(file_path) + character(len=*), intent(in) :: file_path + integer :: unit, ios + open(newunit=unit, file=file_path, status='old', iostat=ios) + if (ios == 0) close(unit, status='delete') + end subroutine delete_file + + !> A one-object float32 descriptor envelope for `n` elements. + function descriptors_json(n) result(s) + integer, intent(in) :: n + character(len=:), allocatable :: s + character(len=32) :: ns + write (ns, '(i0)') n + s = '{"descriptors":[{"type":"ndarray","ndim":1,"shape":[' // trim(ns) // & + '],"strides":[1],"dtype":"float32","byte_order":"' // host_bo() // & + '","encoding":"none","filter":"none","compression":"none"}]}' + end function descriptors_json + + !> Host byte order as the wire descriptor spells it ("little" / "big"). + function host_bo() result(bo) + character(len=:), allocatable :: bo + integer(c_int8_t) :: probe(4) + probe = transfer(1_c_int32_t, 0_c_int8_t, 4) + if (probe(1) == 1_c_int8_t) then + bo = 'little' + else + bo = 'big' + end if + end function host_bo + +end program frame_walker diff --git a/examples/python/21_frames_and_message_header.py b/examples/python/21_frames_and_message_header.py new file mode 100644 index 00000000..4c73b9a4 --- /dev/null +++ b/examples/python/21_frames_and_message_header.py @@ -0,0 +1,186 @@ +# (C) Copyright 2026- ECMWF and individual contributors. +# +# This software is licensed under the terms of the Apache Licence Version 2.0 +# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +# In applying this licence, ECMWF does not waive the privileges and immunities +# granted to it by virtue of its status as an intergovernmental organisation nor +# does it submit to any jurisdiction. + +""" +Example 21 — Walk a message's frames and read its header +======================================================== + +Structural introspection: what is *inside* a message, without decoding +any payload or CBOR. Two entry points, both direct bindings of the Rust +core: + + 1. ``tensogram.message_header(msg)`` — the 24-byte preamble as typed + values: ``version``, ``total_length``, and eight ``has_*`` + predicates that say whether the metadata / index / hash frames live + in the *header* (random-access) or the *footer* (streaming). + + 2. ``tensogram.frames(msg)`` — a lazy iterator over the message's + frames. Each ``Frame`` carries ``frame_type`` (a readable name such + as ``"NTensorFrame"``), ``frame_type_code`` (its numeric wire code), + ``version``, ``flags``, ``offset``, ``length``, ``payload`` and + ``has_hash``. The preamble and postamble are *not* frames. + +Both take **one** message. For a multi-message buffer or file, slice it +with ``tensogram.scan()`` first — shown in ``walk_a_multi_message_buffer``. + +Run: python 21_frames_and_message_header.py +""" + +import numpy as np +import tensogram + +DESCRIPTOR = {"type": "ntensor", "shape": [3, 4], "dtype": "float32"} + + +def make_message(n_objects: int = 2, **encode_kwargs) -> bytes: + """Buffered encode → a random-access message (header-side index).""" + meta = {"base": [{"param": "2t", "step": i} for i in range(n_objects)]} + pairs = [ + (DESCRIPTOR, np.random.default_rng(seed=i).random((3, 4), dtype=np.float32)) + for i in range(n_objects) + ] + return tensogram.encode(meta, pairs, **encode_kwargs) + + +def show_message_header(msg: bytes) -> None: + """Read the envelope — no frame is touched.""" + header = tensogram.message_header(msg) + print("1. message_header:") + print(f" {header!r}") + print(f" version={header.version} total_length={header.total_length}") + print(f" raw flags=0x{header.flags:04x}") + # The eight predicates keep the core's `has_` prefix. + layout = "random-access" if header.has_header_index else "streaming" + print(f" layout: {layout}") + for name in ( + "has_header_metadata", + "has_footer_metadata", + "has_header_index", + "has_footer_index", + "has_header_hashes", + "has_footer_hashes", + "has_preceder_metadata", + "has_hashes_present", + ): + print(f" {name:<24} {getattr(header, name)}") + + +def walk_frames(msg: bytes) -> None: + """Walk every frame: type, span, and content size.""" + print("\n2. frames:") + print(f" {'type':<18} {'code':>4} {'offset':>7} {'length':>7} {'payload':>8} hash") + for frame in tensogram.frames(msg): + print( + f" {frame.frame_type:<18} {frame.frame_type_code:>4} " + f"{frame.offset:>7} {frame.length:>7} {len(frame.payload):>8} {frame.has_hash}" + ) + # `offset` / `length` span the whole frame — header through ENDF. + span = msg[frame.offset : frame.offset + frame.length] + assert span[:2] == b"FR" + assert span[-4:] == b"ENDF" + # `payload` is the *content*: the 16-byte frame header and the + # type-specific footer are excluded. + assert frame.payload == span[16 : 16 + len(frame.payload)] + + # The preamble (24 bytes) and the postamble are not frames. + first = next(tensogram.frames(msg)) + print(f" first frame starts at byte {first.offset} (right after the preamble)") + + +def lazy_and_cheap(msg: bytes) -> None: + """`frames` is an iterator: pull one frame, keep the rest for later.""" + print("\n3. lazy iteration:") + walker = tensogram.frames(msg) + print(f" {walker!r}") + head = next(walker) + print(f" next() → {head!r}") + print(f" {len(walker)} frames still to come: {[f.frame_type for f in walker]}") + + # Data-object frames are the ones carrying tensors. + n_objects = sum(1 for f in tensogram.frames(msg) if f.is_data_object) + print(f" data-object frames: {n_objects}") + + +def compare_buffered_and_streaming() -> None: + """The same content, two layouts — visible in the header and the walk.""" + print("\n4. buffered vs streaming:") + + buffered = make_message(1) + encoder = tensogram.StreamingEncoder({"base": [{"param": "2t"}]}) + encoder.write_preceder({"step": 0}) + encoder.write_object(DESCRIPTOR, np.zeros((3, 4), dtype=np.float32)) + streamed = encoder.finish() + + for label, msg in (("buffered", buffered), ("streaming", streamed)): + header = tensogram.message_header(msg) + types = [f.frame_type for f in tensogram.frames(msg)] + print(f" {label:<10} {types}") + print( + f" {'':<10} header_index={header.has_header_index} " + f"footer_index={header.has_footer_index} " + f"total_length={header.total_length}" + ) + # A streaming message reports total_length = 0 until it is back-filled; + # the walker still stops cleanly at the postamble. + + +def see_the_aggregate_hash_knob() -> None: + """`aggregate_hash=` decides where the hash frame lands — see it.""" + print("\n5. aggregate_hash placement, verified with the walker:") + for policy in ("auto", "none", "header", "footer", "both"): + types = [f.frame_type for f in tensogram.frames(make_message(1, aggregate_hash=policy))] + placed = [t for t in types if t.endswith("Hash")] or ["(none)"] + print(f" aggregate_hash={policy:<8} → {placed}") + + # `compression_backend=` picks the szip / zstd implementation when both + # an FFI and a pure-Rust one are compiled in ("auto", "ffi", "pure"). + msg = make_message(1, compression_backend="pure") + print(f" compression_backend='pure' → {len(tensogram.decode(msg).objects)} object decoded") + + +def walk_a_multi_message_buffer() -> None: + """Slice with `scan` first: both walkers take exactly one message.""" + print("\n6. multi-message buffer:") + buf = make_message(1) + make_message(3) + for i, (offset, length) in enumerate(tensogram.scan(buf)): + msg = buf[offset : offset + length] + types = [f.frame_type for f in tensogram.frames(msg)] + print(f" message[{i}] at {offset:>5} (+{length}): {types}") + + +def report_a_broken_chain() -> None: + """A truncated chain yields the frames that parsed, then raises.""" + print("\n7. a truncated message:") + msg = make_message(2) + last = list(tensogram.frames(msg))[-1] + truncated = msg[: last.offset + 8] # cut inside the final frame + + walker = tensogram.frames(truncated) # the preamble still parses + seen = [] + try: + for frame in walker: + seen.append(frame.frame_type) + except ValueError as exc: + print(f" walked {len(seen)} frames, then: {exc}") + print(f" frames recovered before the break: {seen}") + + +def main() -> None: + msg = make_message(2) + show_message_header(msg) + walk_frames(msg) + lazy_and_cheap(msg) + compare_buffered_and_streaming() + see_the_aggregate_hash_knob() + walk_a_multi_message_buffer() + report_a_broken_chain() + print("\nExample 21 complete.") + + +if __name__ == "__main__": + main() diff --git a/examples/python/README.md b/examples/python/README.md index c617128e..05671c27 100644 --- a/examples/python/README.md +++ b/examples/python/README.md @@ -51,6 +51,7 @@ uv pip install -e python/tensogram-zarr/ | `17_convert_grib.py` | Convert GRIB → Tensogram via `tensogram.convert_grib` (file) and `tensogram.convert_grib_buffer` (in-memory) | | `19_async_streaming_and_common.py` | Async streaming encode with `AsyncStreamingEncoder`, cross-object `compute_common`, and per-object `object_inline_hashes` | | `20_scan_validate_layouts.py` | Reader-side symmetry: `scan_file`/`scan_with_options`/`ScanOptions`, `validate_buffer`, `message_layouts`, lazy `objects`/`objects_metadata`, and `open_mmap` | +| `21_frames_and_message_header.py` | Structural introspection: the `frames()` frame walker and the typed `message_header()`, plus the `aggregate_hash=` / `compression_backend=` encode knobs | For **narrative walk-throughs** with live plots and prose explanations, see the companion notebooks under `../jupyter/`. @@ -59,7 +60,10 @@ see the companion notebooks under `../jupyter/`. ``` tensogram -├── encode(metadata, descriptors_and_data, hash="xxh3") -> bytes +├── encode(metadata, descriptors_and_data, hash="xxh3", +│ aggregate_hash="auto", # or "none"|"header"|"footer"|"both" +│ compression_backend="auto") # or "ffi"|"pure" (szip / zstd) +│ -> bytes ├── decode(buf, verify_hash=False) -> Message ├── decode_metadata(buf) -> Metadata ├── decode_object(buf, index, verify_hash=False) -> (Metadata, DataObjectDescriptor, ndarray) @@ -71,6 +75,8 @@ tensogram ├── scan_file_with_options(path, ScanOptions(...)) -> list[tuple[int, int]] ├── objects(buf, ...) -> ObjectIter # lazy (descriptor, ndarray) ├── objects_metadata(buf) -> ObjectMetadataIter # lazy descriptors only +├── frames(msg) -> FrameIter # lazy Frame walk of ONE message +├── message_header(msg) -> MessageHeader # the preamble, typed ├── object_inline_hashes(buf) -> list[str | None] # per-object xxh3-64 hex, or None ├── iter_messages(buf, verify_hash=False) -> MessageIter ├── compute_packing_params(values, bits_per_value, decimal_scale_factor) -> dict @@ -78,13 +84,18 @@ tensogram ├── validate(buf, level="default", check_canonical=False) -> dict # single message ├── validate_buffer(buf, level="default", check_canonical=False) -> dict # multi-message buffer ├── validate_file(path, level="default", check_canonical=False) -> dict -├── StreamingEncoder(metadata, hash="xxh3") # + write_preceder / object_count() / bytes_written() -├── AsyncStreamingEncoder.create(metadata, hash="xxh3") -> awaitable # async write_object / finish +├── StreamingEncoder(metadata, hash="xxh3", aggregate_hash="auto", compression_backend="auto") +│ # + write_preceder / object_count() / bytes_written() +│ # aggregate_hash "header"/"both" are rejected here +├── AsyncStreamingEncoder.create(metadata, hash="xxh3", ...) -> awaitable +│ # same knobs as StreamingEncoder +│ # async write_object / finish └── TensogramFile ├── open(path) -> TensogramFile ├── open_mmap(path) -> TensogramFile # zero-copy memory-mapped reads (mmap feature) ├── create(path) -> TensogramFile - ├── append(metadata, descriptors_and_data, hash="xxh3") + ├── append(metadata, descriptors_and_data, hash="xxh3", + │ aggregate_hash="auto", compression_backend="auto") ├── message_count() -> int ├── message_layouts() -> list[MessageLayout] # per-message (offset, length) ├── read_message(index) -> bytes @@ -112,6 +123,26 @@ tensogram.DataObjectDescriptor .byte_order, .encoding, .filter, .compression .params -> dict # encoding parameters (e.g. reference_value, bits_per_value) .hash -> None # deprecated in v3; use tensogram.object_inline_hashes(buf) + +tensogram.Frame # yielded by tensogram.frames(msg) + .frame_type -> str # "HeaderMetadata" | "HeaderIndex" | "HeaderHash" + # | "FooterHash" | "FooterIndex" | "FooterMetadata" + # | "PrecederMetadata" | "NTensorFrame" + .frame_type_code -> int # the numeric wire code (1,2,3,5,6,7,8,9) + .version, .flags -> int # frame header fields (flags bit 1 = HASH_PRESENT) + .offset, .length -> int # whole-frame span, relative to the message start + .payload -> bytes # frame content (header + type-specific footer excluded) + .has_hash -> bool # this frame's hash slot holds a real digest + .is_data_object -> bool # True for NTensorFrame + +tensogram.MessageHeader # returned by tensogram.message_header(msg) + .version -> int # wire-format version + .total_length -> int # 0 for a streaming message that was not back-filled + .flags -> int # raw preamble bits; prefer the predicates below + .has_header_metadata, .has_footer_metadata -> bool + .has_header_index, .has_footer_index -> bool + .has_header_hashes, .has_footer_hashes -> bool + .has_preceder_metadata, .has_hashes_present -> bool ``` ## NumPy Integration diff --git a/examples/rust/README.md b/examples/rust/README.md index 2ebe9c05..5da3ff6c 100644 --- a/examples/rust/README.md +++ b/examples/rust/README.md @@ -50,6 +50,7 @@ cargo build --release -p tensogram-rust-examples --features netcdf,remote | `14_remote_access.rs` | Opening a `.tgm` file over HTTP with a self-contained Range-capable server (requires `--features remote`) | | `16_multi_threaded_pipeline.rs` | Caller-controlled `threads=N` encode/decode with determinism invariants | | `18_remote_scan_trace.rs` | Subscribe to `tensogram::remote_scan` tracing events while running forward-only and bidirectional walkers (requires `--features remote`) | +| `19_frame_walker.rs` | Structural introspection: read a message envelope with `message_header()` and walk its frames with `frames()`; contrast buffered vs streaming layout and verify `aggregate_hash` placement | > Two bins share the `11_` prefix (`11_encode_pre_encoded` for the > pre-encoded payload API, `11_streaming` for the progressive diff --git a/examples/rust/src/bin/19_frame_walker.rs b/examples/rust/src/bin/19_frame_walker.rs new file mode 100644 index 00000000..740ec61d --- /dev/null +++ b/examples/rust/src/bin/19_frame_walker.rs @@ -0,0 +1,174 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +//! Example 19 — Frame introspection: walking a message's structure +//! +//! Every .tgm message is a 24-byte preamble, a sequence of frames, and a +//! postamble. `message_header()` reads the envelope and `frames()` walks the +//! frames — both *without* decoding payloads or CBOR, so they are cheap enough +//! to run over a whole file. +//! +//! This example: +//! - Reads the envelope (wire version, total length, structural flags) +//! - Walks the frames of a buffered (random-access) message +//! - Shows that `offset`/`length` tile the message and `payload` is the +//! frame *content* (frame header and type-specific footer stripped) +//! - Contrasts a streaming message, whose index/hashes live in the footer +//! - Uses the walker to *prove* where `aggregate_hash` puts its hash frames +//! +//! Note: `frames()` takes ONE message. For a multi-message buffer, use `scan()` +//! to get each `(offset, length)` and slice first — see example 07. + +use std::collections::BTreeMap; + +use tensogram::{ + ByteOrder, DataObjectDescriptor, Dtype, EncodeOptions, FrameInfo, GlobalMetadata, + StreamingEncoder, encode, frames, message_header, +}; + +fn descriptor(shape: Vec) -> DataObjectDescriptor { + let strides = { + let mut s = vec![1u64; shape.len()]; + for i in (0..shape.len().saturating_sub(1)).rev() { + s[i] = s[i + 1] * shape[i + 1]; + } + s + }; + DataObjectDescriptor { + obj_type: "ntensor".to_string(), + ndim: shape.len() as u64, + shape, + strides, + dtype: Dtype::Float32, + byte_order: ByteOrder::Little, + encoding: "none".to_string(), + filter: "none".to_string(), + compression: "none".to_string(), + params: BTreeMap::new(), + masks: None, + } +} + +/// Print the envelope and the frame table for one message. +fn describe(label: &str, msg: &[u8]) -> Result<(), Box> { + let header = message_header(msg)?; + println!("\n=== {label} ({} bytes) ===", msg.len()); + println!( + " wire version {} total_length {}{}", + header.version, + header.total_length, + if header.total_length == 0 { + " (streaming: never back-filled)" + } else { + "" + } + ); + // `has_header_index` is the reliable random-access discriminator: the + // streaming encoder also writes a HeaderMetadata frame, so + // `has_header_metadata` is true in both modes. + println!( + " layout: {}", + if header.has_header_index() { + "random-access (index in the header)" + } else { + "streaming (index in the footer)" + } + ); + println!( + " metadata hdr/ftr {}/{} index hdr/ftr {}/{} hashes hdr/ftr {}/{} preceder {}", + header.has_header_metadata(), + header.has_footer_metadata(), + header.has_header_index(), + header.has_footer_index(), + header.has_header_hashes(), + header.has_footer_hashes(), + header.has_preceder_metadata(), + ); + + println!( + " {:<20} {:>8} {:>8} {:>9} hash", + "frame type", "offset", "length", "payload" + ); + let mut covered = 0usize; + for frame in frames(msg)? { + let f: FrameInfo = frame?; + println!( + " {:<20} {:>8} {:>8} {:>9} {}", + format!("{:?}", f.frame_type), + f.offset, + f.length, + f.payload.len(), + if f.has_hash() { "yes" } else { "no" } + ); + covered += f.length; + } + println!( + " frames cover {covered} of {} bytes (rest: envelope + padding)", + msg.len() + ); + Ok(()) +} + +fn main() -> Result<(), Box> { + let meta = GlobalMetadata::default(); + let desc = descriptor(vec![4]); + let data = vec![0u8; 16]; + + // ── Buffered encode: the encoder knows everything up front, so the + // metadata/index/hashes go in the HEADER (random access). + let buffered = encode( + &meta, + &[(&desc, &data), (&desc, &data)], + &EncodeOptions::default(), + )?; + describe("buffered (random-access), 2 objects", &buffered)?; + + // ── Streaming encode: the preamble is written before any object, so the + // index/hashes can only go in the FOOTER. + let streamed = { + let mut enc = StreamingEncoder::new( + std::io::Cursor::new(Vec::new()), + &meta, + &EncodeOptions::default(), + )?; + enc.write_object(&desc, &data)?; + enc.finish()?.into_inner() + }; + describe("streaming, 1 object", &streamed)?; + + // ── The walker is the honest way to verify where hashes actually landed. + println!("\n=== aggregate_hash placement (verified by walking) ==="); + for policy in [ + tensogram::AggregateHashPolicy::Header, + tensogram::AggregateHashPolicy::Footer, + tensogram::AggregateHashPolicy::Both, + tensogram::AggregateHashPolicy::None, + ] { + let opts = EncodeOptions { + aggregate_hash: policy, + ..Default::default() + }; + let msg = encode(&meta, &[(&desc, &data)], &opts)?; + let mut header_hash = false; + let mut footer_hash = false; + for frame in frames(&msg)? { + match frame?.frame_type { + tensogram::FrameType::HeaderHash => header_hash = true, + tensogram::FrameType::FooterHash => footer_hash = true, + _ => {} + } + } + println!( + " {:<8} -> header hash: {header_hash:<5} footer hash: {footer_hash}", + format!("{policy:?}") + ); + } + + println!("\nDone."); + Ok(()) +} diff --git a/examples/typescript/21_frame_walker.ts b/examples/typescript/21_frame_walker.ts new file mode 100644 index 00000000..b475ca11 --- /dev/null +++ b/examples/typescript/21_frame_walker.ts @@ -0,0 +1,175 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +/** + * Example 21 — Frame walker + message header (TypeScript) + * + * Structural introspection: look at what a `.tgm` message *is* made of, + * without decoding a single payload or parsing a byte of CBOR. + * + * 1. {@link messageHeader} — the 24-byte preamble as typed values. Tells + * you whether a message is random-access (metadata / index / hashes in + * the *header*) or streaming (in the *footer*) before you read anything + * else. + * + * 2. {@link frames} — the frame chain of ONE message: type, offset, + * length, flags, and the frame's content with the 16-byte frame header + * and the type-specific footer already stripped. The preamble and the + * postamble are NOT frames and never appear. + * + * 3. Multi-message files — `frames` and `messageHeader` are per-message. + * Use {@link scan} to find the boundaries, then slice. + * + * 4. {@link EncodeOptions.aggregateHash} — choose where the aggregate hash + * frame goes, and see the choice on the wire through the walker. + * + * Run with: npx tsx 21_frame_walker.ts + */ + +import { + encode, + frames, + init, + messageHeader, + scan, + type AggregateHashPolicy, + type DataObjectDescriptor, + type Frame, +} from '@ecmwf.int/tensogram'; + +function describe(shape: number[], dtype: DataObjectDescriptor['dtype']): DataObjectDescriptor { + const strides = new Array(shape.length).fill(1); + for (let i = shape.length - 2; i >= 0; i--) strides[i] = strides[i + 1] * shape[i + 1]; + return { + type: 'ntensor', + ndim: shape.length, + shape, + strides, + dtype, + byte_order: 'little', + encoding: 'none', + filter: 'none', + compression: 'none', + }; +} + +/** One tidy line per frame. */ +function render(f: Frame): string { + const span = `${String(f.offset).padStart(5)}..${String(f.offset + f.length).padEnd(5)}`; + return ( + ` ${f.frameType.padEnd(16)} type=${String(f.frameTypeCode).padStart(2)} ` + + `bytes ${span} (${String(f.length).padStart(4)} B) ` + + `payload=${String(f.payload.byteLength).padStart(4)} B ` + + `hash=${f.hasHash ? 'yes' : 'no '}` + ); +} + +function twoObjectMessage(options?: { aggregateHash?: AggregateHashPolicy }): Uint8Array { + return encode( + { base: [{ note: 'temperature' }, { note: 'humidity' }] }, + [ + { descriptor: describe([2, 2], 'float32'), data: new Float32Array([1, 2, 3, 4]) }, + { descriptor: describe([3], 'float64'), data: new Float64Array([10, 20, 30]) }, + ], + options, + ); +} + +async function main(): Promise { + await init(); + + const msg = twoObjectMessage(); + + // ── 1. messageHeader — the envelope, from 24 bytes ─────────────────── + console.log('─── 1. messageHeader(msg) — the message envelope ────────'); + const header = messageHeader(msg); + console.log(` wire version: ${header.version}`); + console.log(` total length: ${header.totalLength} B (buffer is ${msg.byteLength} B)`); + console.log(` header metadata: ${header.hasHeaderMetadata}`); + console.log(` header index: ${header.hasHeaderIndex}`); + console.log(` header hashes: ${header.hasHeaderHashes}`); + console.log(` footer metadata: ${header.hasFooterMetadata}`); + console.log(` footer index: ${header.hasFooterIndex}`); + console.log(` footer hashes: ${header.hasFooterHashes}`); + console.log(` preceder metadata: ${header.hasPrecederMetadata}`); + console.log(` all frames hashed: ${header.hasHashesPresent}`); + // Header-side metadata + index ⇒ this message supports random access. + const randomAccess = header.hasHeaderMetadata && header.hasHeaderIndex; + console.log(` ⇒ layout: ${randomAccess ? 'random-access' : 'streaming'}`); + + // ── 2. frames — the frame chain ────────────────────────────────────── + console.log('\n─── 2. frames(msg) — the frame chain ────────────────────'); + const chain = frames(msg); + for (const f of chain) console.log(render(f)); + console.log( + ` ${chain.length} frames; ` + + `${chain.filter((f) => f.frameType === 'NTensorFrame').length} data object(s).`, + ); + // The preamble (24 B) and postamble (24 B) are not frames. + console.log(` first frame starts at byte ${chain[0].offset} — after the preamble.`); + + // `payload` is the frame CONTENT: the 16-byte frame header and the + // type-specific footer (20 B for NTensorFrame, 12 B otherwise) are + // stripped. It is a COPY on the JS heap, so it stays valid across + // later WASM calls and can be mutated without touching `msg`. + const meta = chain.find((f) => f.frameType === 'HeaderMetadata'); + if (meta) { + const overhead = meta.length - meta.payload.byteLength; + console.log( + ` HeaderMetadata: ${meta.payload.byteLength} B of CBOR + ${overhead} B of framing.`, + ); + } + + // ── 3. Per-message: scan first, then slice ─────────────────────────── + console.log('\n─── 3. Multi-message file — scan() then slice ───────────'); + const other = twoObjectMessage(); + const file = new Uint8Array(msg.byteLength + other.byteLength); + file.set(msg, 0); + file.set(other, msg.byteLength); + + for (const [i, pos] of scan(file).entries()) { + // frames()/messageHeader() take ONE message — slice it out first. + const message = file.subarray(pos.offset, pos.offset + pos.length); + const types = frames(message).map((f) => f.frameType); + console.log( + ` message ${i} @ ${pos.offset} (${pos.length} B): ` + + `v${messageHeader(message).version} [${types.join(' → ')}]`, + ); + } + // Frame offsets are relative to the message, so add the message offset + // to turn one into a file offset. + const second = scan(file)[1]; + const firstFrame = frames(file.subarray(second.offset, second.offset + second.length))[0]; + console.log( + ` message 1's first frame: offset ${firstFrame.offset} in the message, ` + + `${second.offset + firstFrame.offset} in the file.`, + ); + + // ── 4. aggregateHash — placement, observed on the wire ─────────────── + console.log('\n─── 4. aggregateHash — where the digest list lands ──────'); + const policies: AggregateHashPolicy[] = ['auto', 'none', 'header', 'footer', 'both']; + for (const policy of policies) { + const types = frames(twoObjectMessage({ aggregateHash: policy })).map((f) => f.frameType); + const where = [ + types.includes('HeaderHash') ? 'HeaderHash' : null, + types.includes('FooterHash') ? 'FooterHash' : null, + ].filter(Boolean); + console.log( + ` ${policy.padEnd(7)} → ${(where.length ? where.join(' + ') : '(no aggregate frame)').padEnd(24)} ` + + `[${types.join(' → ')}]`, + ); + } + console.log(' ("auto" resolves to a header frame in this buffered encoder.)'); + + console.log('\nFrame walker + message header OK.'); +} + +main().catch((err: unknown) => { + console.error(err); + process.exit(1); +}); diff --git a/examples/typescript/README.md b/examples/typescript/README.md index 3526b56a..e8342f6a 100644 --- a/examples/typescript/README.md +++ b/examples/typescript/README.md @@ -62,3 +62,4 @@ npx tsx 01_encode_decode.ts | [`18_remote_scan_trace.ts`](18_remote_scan_trace.ts) | Intercept `console.debug` to capture `tensogram:scan:*` events from forward-only and bidirectional walkers | | [`19_file_create_masks.ts`](19_file_create_masks.ts) | `TensogramFile.create` empty-file factory + `append` with NaN/Inf mask options (`allowNan` / `allowInf` / `*MaskMethod`), read back with restoration | | [`20_masks_and_iterators.ts`](20_masks_and_iterators.ts) | `decodeWithMasks` (raw NaN / Inf masks) + lazy `objects` / `objectsMetadata` iterators + `scanWithOptions` / `dataObjectInlineHashes` | +| [`21_frame_walker.ts`](21_frame_walker.ts) | `frames` / `messageHeader` — structural introspection of one message (frame chain, offsets, typed envelope), per-message slicing with `scan`, and `aggregateHash` placement seen on the wire | diff --git a/examples/typescript/package.json b/examples/typescript/package.json index a973686d..f678a58d 100644 --- a/examples/typescript/package.json +++ b/examples/typescript/package.json @@ -25,7 +25,8 @@ "17": "tsx 17_remote_s3_signed_fetch.ts", "19": "tsx 19_file_create_masks.ts", "20": "tsx 20_masks_and_iterators.ts", - "all": "npm run 01 && npm run 02 && npm run 02b && npm run 03 && npm run 04 && npm run 05 && npm run 06 && npm run 07 && npm run 08 && npm run 09 && npm run 10 && npm run 11 && npm run 12 && npm run 13 && npm run 14 && npm run 15 && npm run 16 && npm run 17 && npm run 19 && npm run 20" + "21": "tsx 21_frame_walker.ts", + "all": "npm run 01 && npm run 02 && npm run 02b && npm run 03 && npm run 04 && npm run 05 && npm run 06 && npm run 07 && npm run 08 && npm run 09 && npm run 10 && npm run 11 && npm run 12 && npm run 13 && npm run 14 && npm run 15 && npm run 16 && npm run 17 && npm run 19 && npm run 20 && npm run 21" }, "dependencies": { "@ecmwf.int/tensogram": "file:../../typescript" diff --git a/fortran/CMakeLists.txt b/fortran/CMakeLists.txt index 7bd0b98e..6ab9a120 100644 --- a/fortran/CMakeLists.txt +++ b/fortran/CMakeLists.txt @@ -90,13 +90,17 @@ if(TENSOGRAM_FORTRAN_EXAMPLES) add_executable(fortran_masks ${CMAKE_CURRENT_SOURCE_DIR}/../examples/fortran/masks.f90) target_link_libraries(fortran_masks PRIVATE tensogram_f) + + add_executable(fortran_frame_walker + ${CMAKE_CURRENT_SOURCE_DIR}/../examples/fortran/frame_walker.f90) + target_link_libraries(fortran_frame_walker PRIVATE tensogram_f) endif() if(TENSOGRAM_FORTRAN_TESTS) enable_testing() # Positive tests: succeed (exit 0). - foreach(t test_roundtrip test_dtype_rank test_ranks test_file_api test_metadata test_metadata_value test_streaming test_edge_cases test_errors test_symmetry test_wave_c) + foreach(t test_roundtrip test_dtype_rank test_ranks test_file_api test_metadata test_metadata_value test_streaming test_edge_cases test_errors test_symmetry test_wave_c test_frames test_remote test_typed_enums test_encode_options) add_executable(${t} test/${t}.f90) target_link_libraries(${t} PRIVATE tensogram_f) add_test(NAME ${t} COMMAND ${t}) @@ -105,7 +109,7 @@ if(TENSOGRAM_FORTRAN_TESTS) # Run the examples as smoke tests when they are built — otherwise they are # only compile-checked, never executed. Each must exit 0. if(TENSOGRAM_FORTRAN_EXAMPLES) - foreach(ex fortran_encode_decode fortran_file_api fortran_streaming fortran_validate fortran_masks) + foreach(ex fortran_encode_decode fortran_file_api fortran_streaming fortran_validate fortran_masks fortran_frame_walker) add_test(NAME smoke_${ex} COMMAND ${ex}) endforeach() endif() @@ -128,7 +132,7 @@ if(TENSOGRAM_FORTRAN_TESTS) # mode, each asserting the intended abort fired. add_executable(test_guards test/test_guards.f90) target_link_libraries(test_guards PRIVATE tensogram_f) - foreach(mode buffer message file metadata stream check_ctx check_noctx) + foreach(mode buffer message file metadata stream frames check_ctx check_noctx) add_test(NAME guard_${mode} COMMAND test_guards ${mode}) endforeach() set_tests_properties(guard_buffer PROPERTIES PASS_REGULAR_EXPRESSION "tensogram_buffer is non-copyable") @@ -136,6 +140,7 @@ if(TENSOGRAM_FORTRAN_TESTS) set_tests_properties(guard_file PROPERTIES PASS_REGULAR_EXPRESSION "tensogram_file is non-copyable") set_tests_properties(guard_metadata PROPERTIES PASS_REGULAR_EXPRESSION "tensogram_metadata is non-copyable") set_tests_properties(guard_stream PROPERTIES PASS_REGULAR_EXPRESSION "tensogram_streaming_encoder is non-copyable") + set_tests_properties(guard_frames PROPERTIES PASS_REGULAR_EXPRESSION "tensogram_frame_iterator is non-copyable") set_tests_properties(guard_check_ctx PROPERTIES PASS_REGULAR_EXPRESSION "tensogram: guard-context") set_tests_properties(guard_check_noctx PROPERTIES PASS_REGULAR_EXPRESSION "tensogram: ") @@ -210,5 +215,37 @@ if(TENSOGRAM_FORTRAN_TESTS) add_test(NAME value_type_enum_consistency COMMAND ${BASH_PROGRAM} ${CMAKE_CURRENT_SOURCE_DIR}/test/check_value_type_enum.sh ${CMAKE_CURRENT_SOURCE_DIR}/src/tensogram.F90 ${_tgm_header}) + # The C `tgm_frame_type` enum carries the wire format's frame-type field, + # so its source of truth is the Rust core `FrameType` rather than a + # binding-side mirror; guard the generated header directly against it. + set(_tgm_wire ${CMAKE_CURRENT_SOURCE_DIR}/../rust/tensogram/src/wire.rs) + if(EXISTS ${_tgm_wire}) + add_test(NAME frame_type_enum_consistency + COMMAND ${BASH_PROGRAM} ${CMAKE_CURRENT_SOURCE_DIR}/test/check_frame_type_enum.sh + ${_tgm_wire} ${_tgm_header}) + endif() + # The C `tgm_dtype` enum must cover exactly the core `Dtype` variants. + # tensogram.h is a checked-in generated artefact the bindings mirror by + # hand, so a stale or hand-edited copy is what this catches; the Rust + # exhaustive-match unit tests already guard the crate-internal mapping. + set(_tgm_dtype ${CMAKE_CURRENT_SOURCE_DIR}/../rust/tensogram/src/dtype.rs) + if(EXISTS ${_tgm_dtype}) + add_test(NAME dtype_enum_consistency + COMMAND ${BASH_PROGRAM} ${CMAKE_CURRENT_SOURCE_DIR}/test/check_dtype_enum.sh + ${_tgm_dtype} ${_tgm_header}) + endif() + # Same single-source-of-truth guard, one test per VALUED C enum the + # binding mirrors by hand (deferred Wave B: the frame walker's frame + # types, the typed dtype / byte-order codes, and the two encode-option + # policies). check_enum_mirror.sh is check_error_enum.sh generalised over + # the enum prefix — the extraction is identical, only the name differs. + foreach(_enum TGM_FRAME_TYPE TGM_DTYPE TGM_BYTE_ORDER + TGM_AGGREGATE_HASH_POLICY TGM_COMPRESSION_BACKEND) + string(TOLOWER ${_enum} _enum_lc) + string(REGEX REPLACE "^tgm_" "" _enum_lc ${_enum_lc}) + add_test(NAME ${_enum_lc}_mirror_consistency + COMMAND ${BASH_PROGRAM} ${CMAKE_CURRENT_SOURCE_DIR}/test/check_enum_mirror.sh + ${CMAKE_CURRENT_SOURCE_DIR}/src/tensogram.F90 ${_tgm_header} ${_enum}) + endforeach() endif() endif() diff --git a/fortran/src/tensogram.F90 b/fortran/src/tensogram.F90 index 5bc5649c..e09e8a79 100644 --- a/fortran/src/tensogram.F90 +++ b/fortran/src/tensogram.F90 @@ -47,6 +47,15 @@ !> The types are NON-COPYABLE: a defined `assignment(=)` calls `error stop`, !> so an accidental `b = a` aborts loudly at the copy site instead of !> aliasing the handle and double-freeing later. Pass handles by reference. +!> `tensogram_frame_iterator` is the same handle design over the C frame +!> cursor, plus one Fortran-specific twist: the C cursor BORROWS the message +!> bytes for its whole lifetime, which no Fortran dummy argument can promise +!> (a non-contiguous or non-target actual is associated through a compiler +!> temporary that dies at the end of the call), so the iterator keeps a +!> private HEAP COPY of the buffer and lends the cursor that instead. What +!> the walk yields — `tensogram_frame` — is by contrast a plain VALUE with +!> its payload bytes copied out: freely assignable, owning nothing, and +!> valid after both the iterator and the message buffer are gone. module tensogram use, intrinsic :: iso_c_binding use, intrinsic :: iso_fortran_env, only : error_unit @@ -115,6 +124,39 @@ module tensogram public :: tensogram_decode_with_masks public :: tensogram_object_has_masks, tensogram_object_mask public :: TGM_MASK_KIND_NAN, TGM_MASK_KIND_POS_INF, TGM_MASK_KIND_NEG_INF + !> Deferred Wave-B, component A3 — the lazy frame walker (one value type per + !> frame, with the payload bytes COPIED out of the message) and the typed + !> message header (the preamble's structural flags as named logicals). + public :: tensogram_frame, tensogram_frame_iterator, tensogram_frames + public :: tensogram_message_header, tensogram_message_header_read + !> Deferred Wave-B, component B — the SYNCHRONOUS remote surface. The + !> handle returned by tensogram_file_open_remote is an ordinary + !> tensogram_file, so the whole existing file API works against S3 / GCS / + !> Azure / HTTP sources unchanged. + public :: tensogram_is_remote_url, tensogram_file_open_remote + !> Deferred Wave-B, component C — the typed enum surface (the `switch`able + !> companions of the dtype / byte-order STRING getters, which stay). + public :: tensogram_object_dtype_enum, tensogram_object_byte_order_enum + !> …and the full encode-side option set (TgmEncodeOptions): the hash + !> algorithm, WHERE the aggregate hash frame goes, which codec backend to + !> use, plus every mask-companion knob — one entry point per encode target. + public :: tensogram_encode_with_options, tensogram_file_append_with_options + public :: tensogram_streaming_encoder_create_with_options + public :: TGM_FRAME_TYPE_HEADER_METADATA, TGM_FRAME_TYPE_HEADER_INDEX, & + TGM_FRAME_TYPE_HEADER_HASH, TGM_FRAME_TYPE_FOOTER_HASH, & + TGM_FRAME_TYPE_FOOTER_INDEX, TGM_FRAME_TYPE_FOOTER_METADATA, & + TGM_FRAME_TYPE_PRECEDER_METADATA, TGM_FRAME_TYPE_NTENSOR + public :: TGM_DTYPE_FLOAT16, TGM_DTYPE_BFLOAT16, TGM_DTYPE_FLOAT32, & + TGM_DTYPE_FLOAT64, TGM_DTYPE_COMPLEX64, TGM_DTYPE_COMPLEX128, & + TGM_DTYPE_INT8, TGM_DTYPE_INT16, TGM_DTYPE_INT32, TGM_DTYPE_INT64, & + TGM_DTYPE_UINT8, TGM_DTYPE_UINT16, TGM_DTYPE_UINT32, & + TGM_DTYPE_UINT64, TGM_DTYPE_BITMASK + public :: TGM_BYTE_ORDER_LITTLE, TGM_BYTE_ORDER_BIG + public :: TGM_AGGREGATE_HASH_POLICY_AUTO, TGM_AGGREGATE_HASH_POLICY_NONE, & + TGM_AGGREGATE_HASH_POLICY_HEADER, TGM_AGGREGATE_HASH_POLICY_FOOTER, & + TGM_AGGREGATE_HASH_POLICY_BOTH + public :: TGM_COMPRESSION_BACKEND_AUTO, TGM_COMPRESSION_BACKEND_FFI, & + TGM_COMPRESSION_BACKEND_PURE public :: tensogram_strerror, tensogram_last_error, tensogram_check public :: TGM_ERROR_OK, TGM_ERROR_FRAMING, TGM_ERROR_METADATA, & TGM_ERROR_ENCODING, TGM_ERROR_COMPRESSION, TGM_ERROR_OBJECT, & @@ -167,6 +209,75 @@ module tensogram integer(c_int), parameter :: TGM_MASK_KIND_POS_INF = 1 integer(c_int), parameter :: TGM_MASK_KIND_NEG_INF = 2 + ! ---- tgm_frame_type enum — mirror of tensogram.h ------------------------ + ! A frame's type identifier, as reported by tensogram_frame%frame_type(). + ! These numbers ARE the wire format's frame-type field (see + ! plans/WIRE_FORMAT.md §2.2), not an FFI invention: type 4 is RESERVED (it + ! held the obsolete v2 data-object layout) and therefore has no parameter, + ! which is why the sequence skips from 3 to 5. NTENSOR is the only + ! data-object type in v3 and the only one with a 20-byte frame footer + ! (every other type has 12). CI guards this mirror against the C header. + integer(c_int), parameter :: TGM_FRAME_TYPE_HEADER_METADATA = 1 + integer(c_int), parameter :: TGM_FRAME_TYPE_HEADER_INDEX = 2 + integer(c_int), parameter :: TGM_FRAME_TYPE_HEADER_HASH = 3 + integer(c_int), parameter :: TGM_FRAME_TYPE_FOOTER_HASH = 5 + integer(c_int), parameter :: TGM_FRAME_TYPE_FOOTER_INDEX = 6 + integer(c_int), parameter :: TGM_FRAME_TYPE_FOOTER_METADATA = 7 + integer(c_int), parameter :: TGM_FRAME_TYPE_PRECEDER_METADATA = 8 + integer(c_int), parameter :: TGM_FRAME_TYPE_NTENSOR = 9 + + ! ---- tgm_dtype enum — mirror of tensogram.h ----------------------------- + ! An element type, as reported by tensogram_object_dtype_enum. The WIRE + ! stores the dtype as a STRING (plans/WIRE_FORMAT.md §6.1), so unlike the + ! frame types these codes are an FFI convenience rather than a wire value + ! — but they are part of the frozen C ABI and must not be renumbered. + ! tensogram_object_dtype returns the same information as text. Fortran can + ! encode/decode arrays of float32/64 and int32/64 (see tensogram_encode); + ! every other code can still appear on a message written elsewhere. + integer(c_int), parameter :: TGM_DTYPE_FLOAT16 = 0 + integer(c_int), parameter :: TGM_DTYPE_BFLOAT16 = 1 + integer(c_int), parameter :: TGM_DTYPE_FLOAT32 = 2 + integer(c_int), parameter :: TGM_DTYPE_FLOAT64 = 3 + integer(c_int), parameter :: TGM_DTYPE_COMPLEX64 = 4 + integer(c_int), parameter :: TGM_DTYPE_COMPLEX128 = 5 + integer(c_int), parameter :: TGM_DTYPE_INT8 = 6 + integer(c_int), parameter :: TGM_DTYPE_INT16 = 7 + integer(c_int), parameter :: TGM_DTYPE_INT32 = 8 + integer(c_int), parameter :: TGM_DTYPE_INT64 = 9 + integer(c_int), parameter :: TGM_DTYPE_UINT8 = 10 + integer(c_int), parameter :: TGM_DTYPE_UINT16 = 11 + integer(c_int), parameter :: TGM_DTYPE_UINT32 = 12 + integer(c_int), parameter :: TGM_DTYPE_UINT64 = 13 + integer(c_int), parameter :: TGM_DTYPE_BITMASK = 14 + + ! ---- tgm_byte_order enum — mirror of tensogram.h ------------------------ + ! A payload's byte order, as reported by tensogram_object_byte_order_enum; + ! the wire stores "little" / "big" as text (tensogram_object_byte_order). + integer(c_int), parameter :: TGM_BYTE_ORDER_LITTLE = 0 + integer(c_int), parameter :: TGM_BYTE_ORDER_BIG = 1 + + ! ---- tgm_aggregate_hash_policy enum — mirror of tensogram.h ------------- + ! Where the aggregate hash frame goes (tensogram_encode_with_options). + ! AUTO is the zero value: header when buffering, footer when streaming. + ! HEADER and BOTH are BUFFERED-MODE ONLY — a streaming encoder writes its + ! header before any data object exists, so the per-object hashes are not + ! yet known and tensogram_streaming_encoder_create_with_options rejects + ! them with TGM_ERROR_ENCODING. + integer(c_int), parameter :: TGM_AGGREGATE_HASH_POLICY_AUTO = 0 + integer(c_int), parameter :: TGM_AGGREGATE_HASH_POLICY_NONE = 1 + integer(c_int), parameter :: TGM_AGGREGATE_HASH_POLICY_HEADER = 2 + integer(c_int), parameter :: TGM_AGGREGATE_HASH_POLICY_FOOTER = 3 + integer(c_int), parameter :: TGM_AGGREGATE_HASH_POLICY_BOTH = 4 + + ! ---- tgm_compression_backend enum — mirror of tensogram.h --------------- + ! Which codec implementation to prefer for szip / zstd where both are + ! compiled in. AUTO is the zero value (consult TENSOGRAM_COMPRESSION_ + ! BACKEND, else the platform default); FFI and PURE always win over the + ! environment. Purely an implementation choice — the bytes are identical. + integer(c_int), parameter :: TGM_COMPRESSION_BACKEND_AUTO = 0 + integer(c_int), parameter :: TGM_COMPRESSION_BACKEND_FFI = 1 + integer(c_int), parameter :: TGM_COMPRESSION_BACKEND_PURE = 2 + ! ---- Interoperable POD struct: tgm_bytes_t ------------------------------ type, bind(C) :: tgm_bytes_t type(c_ptr) :: data = c_null_ptr @@ -213,6 +324,79 @@ module tensogram integer(c_int64_t) :: small_mask_threshold_bytes = -1_c_int64_t end type tgm_encode_mask_options_t + ! ---- Interoperable POD struct: TgmEncodeOptions ------------------------- + ! The FULL encode-side option set (mirror of the C `TgmEncodeOptions` + ! POD), which supersedes tgm_encode_mask_options_t: the same six mask + ! fields plus the three knobs that previously had no C surface — the hash + ! algorithm, the aggregate-hash placement and the codec backend. + ! Populated internally by the tensogram_*_with_options wrappers from their + ! optional arguments; not part of the public surface (the string fields + ! are C pointers). EVERY field's default below IS the library default, so + ! a default-constructed value behaves exactly like a NULL options pointer: + ! no hashing, AUTO placement, AUTO backend, non-finite input rejected, + ! default mask methods ("roaring") and the default small-mask threshold. + ! `small_mask_threshold_bytes` is a `ptrdiff_t`, which has no F2008 + ! iso_c_binding constant (c_ptrdiff_t is F2018), so it maps onto the + ! pointer-width signed c_int64_t as in tgm_encode_mask_options_t. + type, bind(C) :: tgm_encode_options_t + type(c_ptr) :: hash = c_null_ptr + integer(c_int) :: aggregate_hash = 0_c_int ! ..._AUTO + integer(c_int) :: compression_backend = 0_c_int ! ..._AUTO + logical(c_bool) :: allow_nan = .false._c_bool + logical(c_bool) :: allow_inf = .false._c_bool + type(c_ptr) :: nan_mask_method = c_null_ptr + type(c_ptr) :: pos_inf_mask_method = c_null_ptr + type(c_ptr) :: neg_inf_mask_method = c_null_ptr + integer(c_int64_t) :: small_mask_threshold_bytes = -1_c_int64_t + end type tgm_encode_options_t + + ! ---- Interoperable POD struct: TgmRemoteScanOptions --------------------- + ! Reader-side scan-walker options for tgm_file_open_remote. Populated + ! internally by tensogram_file_open_remote from its optional + ! `bidirectional` argument; not part of the public surface (a one-field + ! struct is better spelled as an optional argument than as a type the + ! caller must declare). The default reproduces the library default, so + ! passing a default-constructed value is equivalent to a NULL pointer. + type, bind(C) :: tgm_remote_scan_options_t + logical(c_bool) :: bidirectional = .true._c_bool + end type tgm_remote_scan_options_t + + ! ---- Interoperable POD struct: TgmFrame --------------------------------- + ! One frame's structural description as filled by tgm_frame_iter_next. The + ! `payload` member BORROWS the message buffer the cursor walks, so this + ! struct never leaves the binding: tensogram_frame (below) copies every + ! field, payload bytes included. `version` / `flags` are u16 on the wire + ! and are carried here as their c_int16_t bit pattern (Fortran has no + ! unsigned kind); u16() widens them back to 0..65535. + type, bind(C) :: tgm_frame_t + integer(c_int) :: frame_type = 0_c_int ! tgm_frame_type + integer(c_int16_t) :: version = 0_c_int16_t ! uint16_t + integer(c_int16_t) :: flags = 0_c_int16_t ! uint16_t + integer(c_size_t) :: offset = 0_c_size_t + integer(c_size_t) :: length = 0_c_size_t + type(c_ptr) :: payload = c_null_ptr ! borrowed, never freed + integer(c_size_t) :: payload_len = 0_c_size_t + end type tgm_frame_t + + ! ---- Interoperable POD struct: TgmMessageHeader ------------------------- + ! A message's envelope (the 24-byte preamble) as filled by + ! tgm_message_header: the wire version, the whole-message byte count (0 + ! when a streaming writer never back-filled it) and the eight structural + ! flags. Copied field-by-field into tensogram_message_header, which + ! presents them as Fortran default logicals. + type, bind(C) :: tgm_message_header_t + integer(c_int16_t) :: version = 0_c_int16_t ! uint16_t + integer(c_int64_t) :: total_length = 0_c_int64_t ! uint64_t + logical(c_bool) :: has_header_metadata = .false._c_bool + logical(c_bool) :: has_footer_metadata = .false._c_bool + logical(c_bool) :: has_header_index = .false._c_bool + logical(c_bool) :: has_footer_index = .false._c_bool + logical(c_bool) :: has_header_hashes = .false._c_bool + logical(c_bool) :: has_footer_hashes = .false._c_bool + logical(c_bool) :: has_preceder_metadata = .false._c_bool + logical(c_bool) :: has_hashes_present = .false._c_bool + end type tgm_message_header_t + ! ---- Owned encoded buffer (RAII over tgm_bytes_t) ----------------------- type :: tensogram_buffer type(tgm_bytes_t), private :: raw @@ -307,6 +491,83 @@ module tensogram final :: stream_enc_final end type tensogram_streaming_encoder + ! ---- One walked frame (a plain VALUE, not a handle) --------------------- + ! Everything tgm_frame_iter_next reports about one frame, COPIED: the C + ! struct's `payload` borrows the message buffer, so a Fortran frame keeps + ! its own `integer(c_int8_t)` copy of those bytes (like %as_bytes on the + ! metadata value cursor). Consequences: a frame is freely copyable and + ! assignable, owns nothing the caller must release, and stays readable + ! after its iterator — and the message buffer — are gone. + type :: tensogram_frame + integer(c_int), private :: ftype = 0_c_int !> TGM_FRAME_TYPE_* + integer, private :: fver = 0 !> u16, widened + integer, private :: fflags = 0 !> u16, widened + integer(c_size_t), private :: foff = 0_c_size_t !> 1-based index + integer(c_size_t), private :: flen = 0_c_size_t + logical, private :: fhash = .false. + integer(c_int8_t), allocatable, private :: fbytes(:) + contains + procedure :: frame_type => frame_frame_type !> TGM_FRAME_TYPE_* code + procedure :: version => frame_version !> frame-header version + procedure :: flags => frame_flags !> raw 16-bit flags + procedure :: offset => frame_offset !> 1-based start in the message + procedure :: length => frame_length !> whole-frame span in bytes + procedure :: payload => frame_payload !> copied content bytes + procedure :: has_hash => frame_has_hash !> HASH_PRESENT (flag bit 1) + end type tensogram_frame + + ! ---- Owned frame cursor (RAII over tgm_frame_iter_t*) ------------------- + ! A lazy walk over one message's frames. The C cursor BORROWS the message + ! bytes for its whole lifetime, which Fortran cannot promise for an + ! arbitrary actual argument (a non-contiguous or non-target actual is + ! associated through a compiler temporary that dies at the end of the + ! call). So tensogram_frames takes a PRIVATE HEAP COPY of the buffer and + ! hands the C cursor that: the borrow is then sound by construction, the + ! caller's array may be modified, deallocated or go out of scope while the + ! walk continues, and every frame's payload has already been copied out + ! anyway. `free` releases both the cursor and the copy (idempotent, and + ! run by the finalizer); the type is NON-COPYABLE like every other handle. + type :: tensogram_frame_iterator + type(c_ptr), private :: ptr = c_null_ptr + integer(c_int8_t), pointer, private :: bytes(:) => null() + contains + procedure :: next => frame_iter_next + procedure :: free => frame_iter_free + procedure, private :: frame_iter_assign + generic, public :: assignment(=) => frame_iter_assign + final :: frame_iter_final + end type tensogram_frame_iterator + + ! ---- A message's envelope (the 24-byte preamble), decoded --------------- + ! The typed companion to the frame walker: the wire version, the whole + ! message byte count and the eight structural flags as named logicals, all + ! without reading a single frame. Together the flags say whether a message + ! is random-access (metadata / index / hashes in the HEADER) or streaming + ! (in the FOOTER). A plain value type: copyable, owns nothing. + type :: tensogram_message_header + integer, private :: ver = 0 + integer(c_int64_t), private :: total = 0_c_int64_t + logical, private :: hmeta = .false. + logical, private :: fmeta = .false. + logical, private :: hindex = .false. + logical, private :: findex = .false. + logical, private :: hhashes = .false. + logical, private :: fhashes = .false. + logical, private :: preceder = .false. + logical, private :: hashes = .false. + contains + procedure :: version => hdr_version + procedure :: total_length => hdr_total_length + procedure :: has_header_metadata => hdr_has_header_metadata + procedure :: has_footer_metadata => hdr_has_footer_metadata + procedure :: has_header_index => hdr_has_header_index + procedure :: has_footer_index => hdr_has_footer_index + procedure :: has_header_hashes => hdr_has_header_hashes + procedure :: has_footer_hashes => hdr_has_footer_hashes + procedure :: has_preceder_metadata => hdr_has_preceder_metadata + procedure :: has_hashes_present => hdr_has_hashes_present + end type tensogram_message_header + ! ---- Generic encode / decode / append over dtype ------------------------ ! Generic over dtype x rank 0..7, expanded from tgm_iface.inc via the ! per-rank table tgm_ranks.inc. The matching @@ -387,6 +648,16 @@ module tensogram #undef FAM end interface tensogram_streaming_encoder_write + ! ---- Remote open, with or without backend storage options --------------- + ! Two specifics rather than optional key/value arrays: parallel arrays are + ! only meaningful together, so "keys without values" is a compile-time + ! error here instead of a runtime one. The specifics are distinguishable + ! by the type of their second argument, and no call can match both. + interface tensogram_file_open_remote + module procedure file_open_remote_plain !> (source, file, err [, bidirectional]) + module procedure file_open_remote_opts !> (source, keys, values, file, err [, bidirectional]) + end interface tensogram_file_open_remote + ! ========================================================================= ! Raw C ABI — synchronous subset of tensogram.h. ! ========================================================================= @@ -1107,6 +1378,134 @@ function c_tgm_encode_with_options(meta, ptrs, lens, n, hash, threads, & type(tgm_bytes_t), intent(out) :: out integer(c_int) :: err end function + + ! ---- Deferred Wave-B, component A3: frame walker + message header ---- + ! The cursor BORROWS `msg` for its whole lifetime (see + ! tensogram_frame_iterator), and each frame's `payload` points into it. + + function c_tgm_frame_iter_create(msg, msg_len) & + bind(C, name="tgm_frame_iter_create") result(it) + import :: c_ptr, c_size_t + type(c_ptr), value :: msg + integer(c_size_t), value :: msg_len + type(c_ptr) :: it ! NULL on a bad preamble + end function + + function c_tgm_frame_iter_next(it, out) & + bind(C, name="tgm_frame_iter_next") result(b) + import :: c_ptr, c_bool, tgm_frame_t + type(c_ptr), value :: it + type(tgm_frame_t), intent(out) :: out + logical(c_bool) :: b ! false = end OR malformed + end function + + subroutine c_tgm_frame_iter_free(it) bind(C, name="tgm_frame_iter_free") + import :: c_ptr + type(c_ptr), value :: it + end subroutine + + function c_tgm_frame_has_hash(frame) & + bind(C, name="tgm_frame_has_hash") result(b) + import :: c_ptr, c_bool + type(c_ptr), value :: frame ! const TgmFrame* + logical(c_bool) :: b + end function + + function c_tgm_message_header(msg, msg_len, out) & + bind(C, name="tgm_message_header") result(err) + import :: c_ptr, c_size_t, c_int, tgm_message_header_t + type(c_ptr), value :: msg + integer(c_size_t), value :: msg_len + type(tgm_message_header_t), intent(out) :: out + integer(c_int) :: err + end function + + ! ---- Deferred Wave-B, component C: the full encode option set -------- + ! NULL options mean the library defaults; the hash algorithm now lives + ! INSIDE the struct, so these have no separate hash_algo argument. + + function c_tgm_encode_with_encode_options(meta, ptrs, lens, n, threads, & + opts, out) bind(C, name="tgm_encode_with_encode_options") result(err) + import :: c_ptr, c_size_t, c_int32_t, c_int, tgm_bytes_t + type(c_ptr), value :: meta + type(c_ptr), value :: ptrs + type(c_ptr), value :: lens + integer(c_size_t), value :: n + integer(c_int32_t), value :: threads + type(c_ptr), value :: opts ! const TgmEncodeOptions* + type(tgm_bytes_t), intent(out) :: out + integer(c_int) :: err + end function + + function c_tgm_file_append_with_encode_options(file, meta, ptrs, lens, n, & + threads, opts) bind(C, name="tgm_file_append_with_encode_options") result(err) + import :: c_ptr, c_size_t, c_int32_t, c_int + type(c_ptr), value :: file + type(c_ptr), value :: meta + type(c_ptr), value :: ptrs + type(c_ptr), value :: lens + integer(c_size_t), value :: n + integer(c_int32_t), value :: threads + type(c_ptr), value :: opts ! const TgmEncodeOptions* + integer(c_int) :: err + end function + + function c_tgm_streaming_encoder_create_with_encode_options(path, meta, & + threads, opts, out) & + bind(C, name="tgm_streaming_encoder_create_with_encode_options") result(err) + import :: c_ptr, c_int32_t, c_int + type(c_ptr), value :: path + type(c_ptr), value :: meta + integer(c_int32_t), value :: threads + type(c_ptr), value :: opts ! const TgmEncodeOptions* + type(c_ptr), intent(out) :: out + integer(c_int) :: err + end function + + ! ---- Deferred Wave-B, component C: typed object accessors ------------ + ! Enum returns have no spare code for failure: a NULL handle or an + ! out-of-range index records the reason in tgm_last_error and returns + ! the ZERO variant, so the wrappers bounds-check before calling. + + function c_tgm_object_dtype_enum(msg, idx) & + bind(C, name="tgm_object_dtype_enum") result(dt) + import :: c_ptr, c_size_t, c_int + type(c_ptr), value :: msg + integer(c_size_t), value :: idx + integer(c_int) :: dt ! tgm_dtype + end function + + function c_tgm_object_byte_order_enum(msg, idx) & + bind(C, name="tgm_object_byte_order_enum") result(bo) + import :: c_ptr, c_size_t, c_int + type(c_ptr), value :: msg + integer(c_size_t), value :: idx + integer(c_int) :: bo ! tgm_byte_order + end function + + ! ---- Deferred Wave-B, component B: synchronous remote ---------------- + ! Always exported: a C library built WITHOUT the `remote` Cargo feature + ! still links, answering .false. / TGM_ERROR_REMOTE with an + ! explanatory tgm_last_error message. + + function c_tgm_is_remote_url(source) & + bind(C, name="tgm_is_remote_url") result(b) + import :: c_ptr, c_bool + type(c_ptr), value :: source + logical(c_bool) :: b + end function + + function c_tgm_file_open_remote(source, keys, values, n_options, opts, out) & + bind(C, name="tgm_file_open_remote") result(err) + import :: c_ptr, c_size_t, c_int + type(c_ptr), value :: source + type(c_ptr), value :: keys ! const char *const * (nullable) + type(c_ptr), value :: values ! const char *const * (nullable) + integer(c_size_t), value :: n_options + type(c_ptr), value :: opts ! const TgmRemoteScanOptions* + type(c_ptr), intent(out) :: out + integer(c_int) :: err + end function end interface contains @@ -1244,6 +1643,122 @@ subroutine resolve_hash(hash, hash_c, hash_ptr) end if end subroutine resolve_hash + !> Marshal a Fortran string array into the C `const char *const *` shape: + !> `flat` is ONE contiguous buffer holding every trimmed string with its + !> NUL terminator, and `ptrs(k)` points at entry k inside it. Both are + !> caller-owned targets that must outlive the C call (the resolve_hash + !> pattern). Trailing blanks — unavoidable in a fixed-length array literal + !> — are dropped, as everywhere else in this binding. A zero-size `strs` + !> still allocates size-1 dummies so nothing ever takes c_loc of a + !> zero-size array; pass NULL alongside n_options = 0 in that case. + subroutine cstr_array(strs, flat, ptrs) + character(len=*), intent(in) :: strs(:) + character(kind=c_char), allocatable, target, intent(out) :: flat(:) + type(c_ptr), allocatable, intent(out) :: ptrs(:) + integer :: n, k, i, total, pos, ln + n = size(strs) + total = 0 + do k = 1, n + total = total + len_trim(strs(k)) + 1 + end do + allocate(flat(max(total, 1)), ptrs(max(n, 1))) + flat = c_null_char + ptrs = c_null_ptr + pos = 1 + do k = 1, n + ln = len_trim(strs(k)) + do i = 1, ln + flat(pos + i - 1) = strs(k)(i:i) + end do + flat(pos + ln) = c_null_char + ptrs(k) = c_loc(flat(pos)) + pos = pos + ln + 1 + end do + end subroutine cstr_array + + !> Resolve an optional mask-method name to a C string pointer. `name_c` is + !> a caller-owned target buffer that must outlive the C call; an absent (or + !> empty) name is NULL, which selects the library default ("roaring"). + subroutine resolve_method(name, name_c, name_ptr) + character(len=*), intent(in), optional :: name + character(kind=c_char), allocatable, target, intent(out) :: name_c(:) + type(c_ptr), intent(out) :: name_ptr + name_ptr = c_null_ptr + if (.not. present(name)) return + if (len_trim(name) == 0) return + call f_to_cstr(trim(name), name_c) + name_ptr = c_loc(name_c) + end subroutine resolve_method + + !> Fill a TgmEncodeOptions POD from the optional arguments the + !> tensogram_*_with_options wrappers share. Every field starts at the + !> library default (see the POD), so only what the caller asked for is + !> touched. The four `*_c` buffers hold the C strings the POD points at and + !> must outlive the C call — they are the caller's targets, exactly as in + !> resolve_hash. `hash` follows the Fortran binding's convention (absent => + !> "xxh3", '' => no hashing), NOT the C ABI's NULL-means-no-hashing. + subroutine resolve_encode_opts(opts, hash_c, nan_c, pos_c, neg_c, hash, & + aggregate_hash, compression_backend, & + allow_nan, allow_inf, nan_mask_method, & + pos_inf_mask_method, neg_inf_mask_method, & + small_mask_threshold_bytes) + type(tgm_encode_options_t), intent(out) :: opts + character(kind=c_char), allocatable, target, intent(out) :: hash_c(:) + character(kind=c_char), allocatable, target, intent(out) :: nan_c(:) + character(kind=c_char), allocatable, target, intent(out) :: pos_c(:) + character(kind=c_char), allocatable, target, intent(out) :: neg_c(:) + character(len=*), intent(in), optional :: hash + integer(c_int), intent(in), optional :: aggregate_hash + integer(c_int), intent(in), optional :: compression_backend + logical, intent(in), optional :: allow_nan, allow_inf + character(len=*), intent(in), optional :: nan_mask_method + character(len=*), intent(in), optional :: pos_inf_mask_method + character(len=*), intent(in), optional :: neg_inf_mask_method + integer(c_int64_t), intent(in), optional :: small_mask_threshold_bytes + call resolve_hash(hash, hash_c, opts%hash) + if (present(aggregate_hash)) opts%aggregate_hash = aggregate_hash + if (present(compression_backend)) opts%compression_backend = compression_backend + if (present(allow_nan)) opts%allow_nan = logical(allow_nan, kind=c_bool) + if (present(allow_inf)) opts%allow_inf = logical(allow_inf, kind=c_bool) + call resolve_method(nan_mask_method, nan_c, opts%nan_mask_method) + call resolve_method(pos_inf_mask_method, pos_c, opts%pos_inf_mask_method) + call resolve_method(neg_inf_mask_method, neg_c, opts%neg_inf_mask_method) + if (present(small_mask_threshold_bytes)) & + opts%small_mask_threshold_bytes = small_mask_threshold_bytes + end subroutine resolve_encode_opts + + !> Marshal concatenated per-object bytes into the C parallel arrays: object + !> k is data(off+1 : off+lens(k)), so num_objects = size(lens) and + !> sum(lens) == size(data). `ptrs` / `lens_c` are caller-owned targets that + !> must outlive the C call; `ptrs_ptr` / `lens_ptr` are NULL for the + !> zero-object case (never take c_loc of a zero-size array). + subroutine object_arrays(data, lens, ptrs, lens_c, ptrs_ptr, lens_ptr) + integer(c_int8_t), target, contiguous, intent(in) :: data(:) + integer(c_size_t), intent(in) :: lens(:) + type(c_ptr), allocatable, target, intent(out) :: ptrs(:) + integer(c_size_t), allocatable, target, intent(out) :: lens_c(:) + type(c_ptr), intent(out) :: ptrs_ptr, lens_ptr + integer(c_size_t) :: off + integer :: nobj, k + nobj = size(lens) + allocate(ptrs(max(nobj, 1)), lens_c(max(nobj, 1))) + ptrs = c_null_ptr + lens_c = 0_c_size_t + off = 0_c_size_t + do k = 1, nobj + if (lens(k) > 0_c_size_t) ptrs(k) = c_loc(data(off + 1_c_size_t)) + lens_c(k) = lens(k) + off = off + lens(k) + end do + if (nobj > 0) then + ptrs_ptr = c_loc(ptrs) + lens_ptr = c_loc(lens_c) + else + ptrs_ptr = c_null_ptr + lens_ptr = c_null_ptr + end if + end subroutine object_arrays + !> Total element count for a Fortran extent vector (1 for a scalar / empty). pure function num_elements(ext) result(n) integer(c_int64_t), intent(in) :: ext(:) @@ -1618,6 +2133,112 @@ subroutine tensogram_encode_with_masks(metadata_json, data, lens, buf, err, & int(nobj, c_size_t), hash_ptr, 0_c_int32_t, c_loc(mopts), buf%raw) end subroutine tensogram_encode_with_masks + !> Encode a message with the FULL encode-side option set (binds + !> tgm_encode_with_encode_options). Supersedes tensogram_encode_with_masks, + !> which stays: this entry point carries the same mask knobs PLUS the three + !> that had no Fortran surface at all — the hash algorithm, WHERE the + !> aggregate hash frame goes, and which codec backend to use. Every + !> argument after `err` is optional and defaults to the library default, so + !> omitting them all is a plain encode. + !> + !> `metadata_json` is the full `{"descriptors":[...]}` envelope (one + !> descriptor per object, same schema as tensogram_encode). `data` is every + !> object's RAW element bytes CONCATENATED; `lens(k)` is object k's byte + !> length, so num_objects = size(lens) and sum(lens) == size(data). + !> + !> hash "xxh3" (default) or '' for no hashing + !> aggregate_hash TGM_AGGREGATE_HASH_POLICY_* placement; the + !> default AUTO puts it in the header here + !> (buffered mode), BOTH writes it twice + !> compression_backend TGM_COMPRESSION_BACKEND_* codec choice — + !> an implementation detail, not a format one + !> allow_nan / allow_inf substitute non-finite values and record + !> companion masks (default: reject) + !> *_mask_method "none"/"rle"/"roaring"/"lz4"/"zstd"/"blosc2" + !> small_mask_threshold_bytes raw-storage threshold; 0 disables the + !> auto-fallback, negative selects the default + subroutine tensogram_encode_with_options(metadata_json, data, lens, buf, err, & + hash, aggregate_hash, & + compression_backend, & + allow_nan, allow_inf, & + nan_mask_method, pos_inf_mask_method,& + neg_inf_mask_method, & + small_mask_threshold_bytes) + character(len=*), intent(in) :: metadata_json + integer(c_int8_t), target, contiguous, intent(in) :: data(:) + integer(c_size_t), intent(in) :: lens(:) + type(tensogram_buffer), intent(out) :: buf + integer(c_int), intent(out) :: err + character(len=*), intent(in), optional :: hash + integer(c_int), intent(in), optional :: aggregate_hash + integer(c_int), intent(in), optional :: compression_backend + logical, intent(in), optional :: allow_nan, allow_inf + character(len=*), intent(in), optional :: nan_mask_method + character(len=*), intent(in), optional :: pos_inf_mask_method + character(len=*), intent(in), optional :: neg_inf_mask_method + integer(c_int64_t), intent(in), optional :: small_mask_threshold_bytes + character(kind=c_char), allocatable, target :: meta_c(:), hash_c(:) + character(kind=c_char), allocatable, target :: nan_c(:), pos_c(:), neg_c(:) + type(c_ptr), allocatable, target :: ptrs(:) + integer(c_size_t), allocatable, target :: lens_c(:) + type(tgm_encode_options_t), target :: opts + type(c_ptr) :: ptrs_ptr, lens_ptr + call f_to_cstr(metadata_json, meta_c) + call resolve_encode_opts(opts, hash_c, nan_c, pos_c, neg_c, hash, & + aggregate_hash, compression_backend, & + allow_nan, allow_inf, nan_mask_method, & + pos_inf_mask_method, neg_inf_mask_method, & + small_mask_threshold_bytes) + call object_arrays(data, lens, ptrs, lens_c, ptrs_ptr, lens_ptr) + err = c_tgm_encode_with_encode_options(c_loc(meta_c), ptrs_ptr, lens_ptr, & + size(lens, kind=c_size_t), 0_c_int32_t, c_loc(opts), buf%raw) + end subroutine tensogram_encode_with_options + + !> Encode a message with the full option set and APPEND it to `file` (binds + !> tgm_file_append_with_encode_options). Same options and same + !> `metadata_json` / `data` / `lens` contract as + !> tensogram_encode_with_options, but the bytes go straight to the open + !> file instead of into a buffer — the raw-bytes counterpart of the generic + !> tensogram_file_append. + subroutine tensogram_file_append_with_options(file, metadata_json, data, lens, err, & + hash, aggregate_hash, & + compression_backend, & + allow_nan, allow_inf, & + nan_mask_method, & + pos_inf_mask_method, & + neg_inf_mask_method, & + small_mask_threshold_bytes) + type(tensogram_file), intent(in) :: file + character(len=*), intent(in) :: metadata_json + integer(c_int8_t), target, contiguous, intent(in) :: data(:) + integer(c_size_t), intent(in) :: lens(:) + integer(c_int), intent(out) :: err + character(len=*), intent(in), optional :: hash + integer(c_int), intent(in), optional :: aggregate_hash + integer(c_int), intent(in), optional :: compression_backend + logical, intent(in), optional :: allow_nan, allow_inf + character(len=*), intent(in), optional :: nan_mask_method + character(len=*), intent(in), optional :: pos_inf_mask_method + character(len=*), intent(in), optional :: neg_inf_mask_method + integer(c_int64_t), intent(in), optional :: small_mask_threshold_bytes + character(kind=c_char), allocatable, target :: meta_c(:), hash_c(:) + character(kind=c_char), allocatable, target :: nan_c(:), pos_c(:), neg_c(:) + type(c_ptr), allocatable, target :: ptrs(:) + integer(c_size_t), allocatable, target :: lens_c(:) + type(tgm_encode_options_t), target :: opts + type(c_ptr) :: ptrs_ptr, lens_ptr + call f_to_cstr(metadata_json, meta_c) + call resolve_encode_opts(opts, hash_c, nan_c, pos_c, neg_c, hash, & + aggregate_hash, compression_backend, & + allow_nan, allow_inf, nan_mask_method, & + pos_inf_mask_method, neg_inf_mask_method, & + small_mask_threshold_bytes) + call object_arrays(data, lens, ptrs, lens_c, ptrs_ptr, lens_ptr) + err = c_tgm_file_append_with_encode_options(file%ptr, c_loc(meta_c), & + ptrs_ptr, lens_ptr, size(lens, kind=c_size_t), 0_c_int32_t, & + c_loc(opts)) + end subroutine tensogram_file_append_with_options + ! ========================================================================= ! Decode wire bytes -> message handle ! ========================================================================= @@ -1845,6 +2466,45 @@ function tensogram_object_byte_order(msg, iobj) result(s) s = cptr_to_fstr(c_tgm_object_byte_order(msg%ptr, int(iobj - 1, c_size_t))) end function tensogram_object_byte_order + !> dtype of object `iobj` (1-based) as a TGM_DTYPE_* code — the typed + !> companion of tensogram_object_dtype, for a SELECT CASE instead of a + !> string comparison. An enum return has no spare code for failure, so an + !> out-of-range `iobj` (or a null handle) yields the zero variant + !> TGM_DTYPE_FLOAT16 and sets the optional `err` to TGM_ERROR_INVALID_ARG — + !> the unambiguous way to bounds-check (tensogram_object_dtype returns '' + !> for exactly the same inputs). + function tensogram_object_dtype_enum(msg, iobj, err) result(dt) + type(tensogram_message), intent(in) :: msg + integer, intent(in) :: iobj + integer(c_int), intent(out), optional :: err + integer(c_int) :: dt + if (iobj < 1 .or. iobj > tensogram_num_objects(msg)) then + if (present(err)) err = TGM_ERROR_INVALID_ARG + dt = TGM_DTYPE_FLOAT16 ! the zero variant, as in C + return + end if + if (present(err)) err = TGM_ERROR_OK + dt = c_tgm_object_dtype_enum(msg%ptr, int(iobj - 1, c_size_t)) + end function tensogram_object_dtype_enum + + !> Byte order of object `iobj` (1-based) as a TGM_BYTE_ORDER_* code — the + !> typed companion of tensogram_object_byte_order. Out-of-range handling + !> matches tensogram_object_dtype_enum: the zero variant + !> TGM_BYTE_ORDER_LITTLE plus TGM_ERROR_INVALID_ARG in the optional `err`. + function tensogram_object_byte_order_enum(msg, iobj, err) result(bo) + type(tensogram_message), intent(in) :: msg + integer, intent(in) :: iobj + integer(c_int), intent(out), optional :: err + integer(c_int) :: bo + if (iobj < 1 .or. iobj > tensogram_num_objects(msg)) then + if (present(err)) err = TGM_ERROR_INVALID_ARG + bo = TGM_BYTE_ORDER_LITTLE ! the zero variant, as in C + return + end if + if (present(err)) err = TGM_ERROR_OK + bo = c_tgm_object_byte_order_enum(msg%ptr, int(iobj - 1, c_size_t)) + end function tensogram_object_byte_order_enum + !> Filter string of object `iobj` (e.g. "none", "shuffle"); '' out of range. function tensogram_object_filter(msg, iobj) result(s) type(tensogram_message), intent(in) :: msg @@ -2102,6 +2762,84 @@ subroutine tensogram_file_create(path, file, err) if (err == TGM_ERROR_OK) file%ptr = out end subroutine tensogram_file_create + !> .true. when `source` is a URL THIS BUILD can open remotely. The + !> recognised schemes are s3, s3a, gs, az, azure, http and https, compared + !> case-insensitively; plain paths and `file://` URLs are NOT remote — they + !> belong to tensogram_file_open. A C library built without the `remote` + !> Cargo feature answers .false. for EVERY input (it genuinely cannot open + !> any remote URL) and records why in tensogram_last_error(). + function tensogram_is_remote_url(source) result(is_remote) + character(len=*), intent(in) :: source + logical :: is_remote + character(kind=c_char), allocatable, target :: source_c(:) + call f_to_cstr(trim(source), source_c) + is_remote = logical(c_tgm_is_remote_url(c_loc(source_c))) + end function tensogram_is_remote_url + + !> Open a remote `.tgm` (S3 / GCS / Azure / HTTP) for SYNCHRONOUS reading. + !> `file` receives an ordinary tensogram_file, so the whole existing file + !> API — message_count / read_message / decode_message — works unchanged; + !> close it with file%close() as usual. + !> + !> `keys` / `values` are parallel arrays of backend storage options + !> (credentials, region, endpoint, …) forwarded verbatim to the object-store + !> backend; call the 3-argument form (source, file, err) when there are + !> none. `bidirectional` (default .true.) selects the meet-in-the-middle + !> remote scan walk; .false. forces a forward-only walk. + !> + !> `err` is TGM_ERROR_INVALID_ARG when `keys` and `values` have different + !> lengths (the C ABI takes ONE option count for both), and TGM_ERROR_REMOTE + !> for an unparseable URL, a missing object, a rejected storage option or a + !> transport failure — including EVERY call on a C library built without the + !> `remote` Cargo feature, whose tensogram_last_error() says how to enable + !> it. Argument validation runs before that feature check, so a mistake is + !> reported as a mistake in either build. + subroutine file_open_remote_opts(source, keys, values, file, err, bidirectional) + character(len=*), intent(in) :: source + character(len=*), intent(in) :: keys(:) + character(len=*), intent(in) :: values(:) + type(tensogram_file), intent(out) :: file + integer(c_int), intent(out) :: err + logical, intent(in), optional :: bidirectional + character(kind=c_char), allocatable, target :: source_c(:), kflat(:), vflat(:) + type(c_ptr), allocatable, target :: kptrs(:), vptrs(:) + type(tgm_remote_scan_options_t), target :: opts + type(c_ptr) :: keys_ptr, values_ptr, out + integer(c_size_t) :: n + if (size(keys) /= size(values)) then + err = TGM_ERROR_INVALID_ARG ! parallel arrays must pair up + return + end if + call f_to_cstr(trim(source), source_c) + call cstr_array(keys, kflat, kptrs) + call cstr_array(values, vflat, vptrs) + n = size(keys, kind=c_size_t) + if (n > 0_c_size_t) then + keys_ptr = c_loc(kptrs) + values_ptr = c_loc(vptrs) + else + keys_ptr = c_null_ptr ! n_options = 0 ignores both arrays + values_ptr = c_null_ptr + end if + ! opts starts at the library defaults, so passing it always is + ! equivalent to passing NULL; flip only what the caller asked for. + if (present(bidirectional)) opts%bidirectional = logical(bidirectional, kind=c_bool) + err = c_tgm_file_open_remote(c_loc(source_c), keys_ptr, values_ptr, n, & + c_loc(opts), out) + if (err == TGM_ERROR_OK) file%ptr = out + end subroutine file_open_remote_opts + + !> Open a remote `.tgm` with no backend storage options — see + !> file_open_remote_opts for the full contract. + subroutine file_open_remote_plain(source, file, err, bidirectional) + character(len=*), intent(in) :: source + type(tensogram_file), intent(out) :: file + integer(c_int), intent(out) :: err + logical, intent(in), optional :: bidirectional + character(len=1) :: none(0) + call file_open_remote_opts(source, none, none, file, err, bidirectional) + end subroutine file_open_remote_plain + !> Number of messages in the file (may trigger a lazy scan). subroutine tensogram_file_message_count(file, count, err) type(tensogram_file), intent(in) :: file @@ -2736,6 +3474,60 @@ subroutine tensogram_streaming_encoder_create(path, enc, err, metadata_json, has if (err == TGM_ERROR_OK) enc%ptr = out end subroutine tensogram_streaming_encoder_create + !> Open a streaming encoder with the FULL encode-side option set (binds + !> tgm_streaming_encoder_create_with_encode_options). Like + !> tensogram_streaming_encoder_create, but the options also carry the + !> aggregate-hash placement and the codec backend — see + !> tensogram_encode_with_options for the shared argument contract. + !> + !> TGM_AGGREGATE_HASH_POLICY_HEADER and ..._BOTH are REJECTED here with + !> TGM_ERROR_ENCODING (and an explanatory tensogram_last_error): a + !> streaming writer emits its header before any data object exists, so the + !> per-object hashes are not yet known. Use AUTO — which resolves to the + !> footer when streaming — or FOOTER. + subroutine tensogram_streaming_encoder_create_with_options(path, enc, err, & + metadata_json, hash, & + aggregate_hash, & + compression_backend, & + allow_nan, allow_inf, & + nan_mask_method, & + pos_inf_mask_method, & + neg_inf_mask_method, & + small_mask_threshold_bytes) + character(len=*), intent(in) :: path + type(tensogram_streaming_encoder), intent(out) :: enc + integer(c_int), intent(out) :: err + character(len=*), intent(in), optional :: metadata_json + character(len=*), intent(in), optional :: hash + integer(c_int), intent(in), optional :: aggregate_hash + integer(c_int), intent(in), optional :: compression_backend + logical, intent(in), optional :: allow_nan, allow_inf + character(len=*), intent(in), optional :: nan_mask_method + character(len=*), intent(in), optional :: pos_inf_mask_method + character(len=*), intent(in), optional :: neg_inf_mask_method + integer(c_int64_t), intent(in), optional :: small_mask_threshold_bytes + character(kind=c_char), allocatable, target :: path_c(:), meta_c(:), hash_c(:) + character(kind=c_char), allocatable, target :: nan_c(:), pos_c(:), neg_c(:) + character(len=:), allocatable :: meta_s + type(tgm_encode_options_t), target :: opts + type(c_ptr) :: out + call f_to_cstr(trim(path), path_c) + if (present(metadata_json)) then + meta_s = metadata_json + else + meta_s = '{}' + end if + call f_to_cstr(meta_s, meta_c) + call resolve_encode_opts(opts, hash_c, nan_c, pos_c, neg_c, hash, & + aggregate_hash, compression_backend, & + allow_nan, allow_inf, nan_mask_method, & + pos_inf_mask_method, neg_inf_mask_method, & + small_mask_threshold_bytes) + err = c_tgm_streaming_encoder_create_with_encode_options(c_loc(path_c), & + c_loc(meta_c), 0_c_int32_t, c_loc(opts), out) + if (err == TGM_ERROR_OK) enc%ptr = out + end subroutine tensogram_streaming_encoder_create_with_options + !> Finalise the stream: write the footer index/hash + postamble and close !> the file. The handle remains valid (release it with `enc%free()`). subroutine tensogram_streaming_encoder_finish(enc, err) @@ -3121,4 +3913,296 @@ function tensogram_compute_hash(data, algo, err) result(hex) end if end function tensogram_compute_hash + ! ========================================================================= + ! Frame walker + message header (deferred Wave-B, component A3) + ! ========================================================================= + + !> Widen a u16 wire field carried as its signed c_int16_t bit pattern to a + !> default integer in 0..65535 (Fortran has no unsigned kind), so bit tests + !> on %flags() and comparisons on %version() read naturally. + pure function u16(x) result(v) + integer(c_int16_t), intent(in) :: x + integer :: v + v = int(x) + if (v < 0) v = v + 65536 + end function u16 + + ! ---- tensogram_frame accessors (pure value reads) ----------------------- + + !> Which kind of frame this is, as a TGM_FRAME_TYPE_* code. + function frame_frame_type(self) result(t) + class(tensogram_frame), intent(in) :: self + integer(c_int) :: t + t = self%ftype + end function frame_frame_type + + !> Frame-type-specific version field from the 16-byte frame header. + function frame_version(self) result(v) + class(tensogram_frame), intent(in) :: self + integer :: v + v = self%fver + end function frame_version + + !> Raw 16-bit frame flags (0..65535); bit 1 is HASH_PRESENT — prefer + !> %has_hash(), which asks the library rather than the bit. + function frame_flags(self) result(f) + class(tensogram_frame), intent(in) :: self + integer :: f + f = self%fflags + end function frame_flags + + !> 1-BASED index of the frame header within the walked message buffer (the + !> C ABI's 0-based byte offset + 1), matching the tensogram_scan + !> convention: the frame occupies buffer(offset : offset + length - 1). + function frame_offset(self) result(o) + class(tensogram_frame), intent(in) :: self + integer(c_size_t) :: o + o = self%foff + end function frame_offset + + !> Whole-frame span in bytes: frame header through the ENDF sentinel, + !> excluding any alignment padding that follows. + function frame_length(self) result(n) + class(tensogram_frame), intent(in) :: self + integer(c_size_t) :: n + n = self%flen + end function frame_length + + !> The frame's content bytes — everything between the 16-byte frame header + !> and the type-specific footer (20 bytes for TGM_FRAME_TYPE_NTENSOR, 12 + !> for every other type) — as an independent COPY, so it outlives both the + !> iterator and the message buffer it was walked from. Size 0 for a frame + !> with no content (and for a default-initialised frame). + function frame_payload(self) result(p) + class(tensogram_frame), intent(in) :: self + integer(c_int8_t), allocatable :: p(:) + if (allocated(self%fbytes)) then + p = self%fbytes + else + allocate(p(0)) + end if + end function frame_payload + + !> .true. when this frame's HASH_PRESENT flag is set, i.e. its inline hash + !> slot holds a meaningful digest (plans/WIRE_FORMAT.md §2.5). Authoritative + !> per frame — tensogram_message_header%has_hashes_present() is only an + !> advisory message-wide summary. + function frame_has_hash(self) result(h) + class(tensogram_frame), intent(in) :: self + logical :: h + h = self%fhash + end function frame_has_hash + + ! ---- tensogram_frame_iterator methods ----------------------------------- + + !> Release the C cursor and the private message copy (idempotent). + subroutine frame_iter_free(self) + class(tensogram_frame_iterator), intent(inout) :: self + if (c_associated(self%ptr)) then + call c_tgm_frame_iter_free(self%ptr) ! cursor first: it borrows bytes + self%ptr = c_null_ptr + end if + if (associated(self%bytes)) then + deallocate(self%bytes) + nullify(self%bytes) + end if + end subroutine frame_iter_free + + subroutine frame_iter_final(self) + type(tensogram_frame_iterator), intent(inout) :: self + call self%free() + end subroutine frame_iter_final + + subroutine frame_iter_assign(lhs, rhs) + class(tensogram_frame_iterator), intent(out) :: lhs + type(tensogram_frame_iterator), intent(in) :: rhs + if (c_associated(rhs%ptr)) then + error stop "tensogram_frame_iterator is non-copyable: a live cursor would alias and double-free; pass by reference" + else + error stop "tensogram_frame_iterator is non-copyable: do not assign cursors; pass by reference" + end if + end subroutine frame_iter_assign + + !> Advance the walk. `found` is .true. when `frame` was filled; .false. + !> ends the loop for BOTH of the C ABI's stop conditions, which the + !> optional `err` tells apart (it is the ONLY place they differ): + !> * clean end — every frame yielded; err = TGM_ERROR_OK + !> * malformed chain — truncated / inconsistent framing; err = + !> TGM_ERROR_FRAMING, with the reason in tensogram_last_error() + !> * dead cursor — %next after %free (or before tensogram_frames); + !> err = TGM_ERROR_INVALID_ARG + !> Calling %next again after any .false. is safe and keeps returning + !> .false.. Omitting `err` collapses the three into "the walk stopped". + subroutine frame_iter_next(self, frame, found, err) + class(tensogram_frame_iterator), intent(inout) :: self + type(tensogram_frame), intent(out) :: frame + logical, intent(out) :: found + integer(c_int), optional, intent(out) :: err + type(tgm_frame_t), target :: raw + integer(c_int8_t), pointer :: view(:) + integer(c_int) :: e + found = .false. + e = TGM_ERROR_OK + if (.not. c_associated(self%ptr)) then + e = TGM_ERROR_INVALID_ARG ! freed / never created + else if (c_tgm_frame_iter_next(self%ptr, raw)) then + found = .true. + frame%ftype = raw%frame_type + frame%fver = u16(raw%version) + frame%fflags = u16(raw%flags) + frame%foff = raw%offset + 1_c_size_t ! 1-based Fortran index + frame%flen = raw%length + frame%fhash = logical(c_tgm_frame_has_hash(c_loc(raw))) + allocate(frame%fbytes(raw%payload_len)) + if (raw%payload_len > 0_c_size_t .and. c_associated(raw%payload)) then + call c_f_pointer(raw%payload, view, [raw%payload_len]) + frame%fbytes = view ! COPY out of the message + end if + else if (c_associated(c_tgm_last_error())) then + e = TGM_ERROR_FRAMING ! malformed: the C layer said why + end if ! else: clean end (error cleared) + if (present(err)) err = e + end subroutine frame_iter_next + + !> Start a lazy walk over the frames of ONE message. `buffer` must start at + !> a message preamble (the `TENSOGRM` magic) — for a multi-message buffer, + !> locate the boundaries with tensogram_scan first and pass each slice. + !> Only the type 1-9 frames are yielded; the preamble and postamble are not + !> frames (see tensogram_message_header_read for the envelope). + !> + !> The iterator takes its OWN copy of `buffer`, so the caller's array may + !> be modified or deallocated while the walk continues. Release the + !> iterator with %free() (or let it be finalized). + !> + !> `err` is TGM_ERROR_OK when the walk started, TGM_ERROR_INVALID_ARG for + !> an empty buffer, and TGM_ERROR_FRAMING when the preamble does not parse + !> (truncated buffer, wrong magic, unsupported version) — the reason is in + !> tensogram_last_error(). + subroutine tensogram_frames(buffer, iterator, err) + integer(c_int8_t), intent(in) :: buffer(:) + type(tensogram_frame_iterator), intent(out) :: iterator + integer(c_int), intent(out) :: err + integer(c_size_t) :: n + type(c_ptr) :: it + n = size(buffer, kind=c_size_t) + if (n == 0_c_size_t) then + err = TGM_ERROR_INVALID_ARG ! no preamble to parse + return + end if + allocate(iterator%bytes(n)) + iterator%bytes = buffer ! private copy: the C cursor + it = c_tgm_frame_iter_create(c_loc(iterator%bytes), n) ! borrows it + if (.not. c_associated(it)) then + deallocate(iterator%bytes) + nullify(iterator%bytes) + err = TGM_ERROR_FRAMING + return + end if + iterator%ptr = it + err = TGM_ERROR_OK + end subroutine tensogram_frames + + ! ---- tensogram_message_header accessors --------------------------------- + + !> Wire-format version of the message (TGM_WIRE_VERSION for v3 messages). + function hdr_version(self) result(v) + class(tensogram_message_header), intent(in) :: self + integer :: v + v = self%ver + end function hdr_version + + !> Whole-message byte count, preamble through postamble, or 0 when a + !> streaming writer never back-filled it (not an error — "unknown at write + !> time"). A u64 on the wire, carried as its signed c_int64_t bit pattern. + function hdr_total_length(self) result(n) + class(tensogram_message_header), intent(in) :: self + integer(c_int64_t) :: n + n = self%total + end function hdr_total_length + + !> A HeaderMetadata frame is present (random-access mode). + function hdr_has_header_metadata(self) result(b) + class(tensogram_message_header), intent(in) :: self + logical :: b + b = self%hmeta + end function hdr_has_header_metadata + + !> A FooterMetadata frame is present (streaming mode). + function hdr_has_footer_metadata(self) result(b) + class(tensogram_message_header), intent(in) :: self + logical :: b + b = self%fmeta + end function hdr_has_footer_metadata + + !> A HeaderIndex frame is present. + function hdr_has_header_index(self) result(b) + class(tensogram_message_header), intent(in) :: self + logical :: b + b = self%hindex + end function hdr_has_header_index + + !> A FooterIndex frame is present. + function hdr_has_footer_index(self) result(b) + class(tensogram_message_header), intent(in) :: self + logical :: b + b = self%findex + end function hdr_has_footer_index + + !> A HeaderHash (aggregate hash) frame is present. + function hdr_has_header_hashes(self) result(b) + class(tensogram_message_header), intent(in) :: self + logical :: b + b = self%hhashes + end function hdr_has_header_hashes + + !> A FooterHash (aggregate hash) frame is present. + function hdr_has_footer_hashes(self) result(b) + class(tensogram_message_header), intent(in) :: self + logical :: b + b = self%fhashes + end function hdr_has_footer_hashes + + !> At least one PrecederMetadata frame appears in the body. ADVISORY in + !> streaming mode: the encoder sets the flag before it knows whether any + !> preceder will be written, so .true. does not guarantee a frame (only + !> "frame present => flag set" holds); in buffered mode it is exact. + function hdr_has_preceder_metadata(self) result(b) + class(tensogram_message_header), intent(in) :: self + logical :: b + b = self%preceder + end function hdr_has_preceder_metadata + + !> ADVISORY: every frame in this message has its per-frame HASH_PRESENT bit + !> set. For any single frame, tensogram_frame%has_hash() stays authoritative. + function hdr_has_hashes_present(self) result(b) + class(tensogram_message_header), intent(in) :: self + logical :: b + b = self%hashes + end function hdr_has_hashes_present + + !> Read a message's envelope (the 24-byte preamble) WITHOUT walking its + !> frames. `buffer` must start at a message preamble, as for + !> tensogram_frames. `err` is TGM_ERROR_OK on success, TGM_ERROR_INVALID_ARG + !> for an empty buffer, or the mapped parse error (message truncated, wrong + !> magic, unsupported version) with the reason in tensogram_last_error(); + !> `header` is left default-initialised on any failure. + subroutine tensogram_message_header_read(buffer, header, err) + integer(c_int8_t), target, contiguous, intent(in) :: buffer(:) + type(tensogram_message_header), intent(out) :: header + integer(c_int), intent(out) :: err + type(tgm_message_header_t) :: raw + err = c_tgm_message_header(buf_ptr(buffer), size(buffer, kind=c_size_t), raw) + if (err /= TGM_ERROR_OK) return + header%ver = u16(raw%version) + header%total = raw%total_length + header%hmeta = logical(raw%has_header_metadata) + header%fmeta = logical(raw%has_footer_metadata) + header%hindex = logical(raw%has_header_index) + header%findex = logical(raw%has_footer_index) + header%hhashes = logical(raw%has_header_hashes) + header%fhashes = logical(raw%has_footer_hashes) + header%preceder = logical(raw%has_preceder_metadata) + header%hashes = logical(raw%has_hashes_present) + end subroutine tensogram_message_header_read + end module tensogram diff --git a/fortran/test/check_dtype_enum.sh b/fortran/test/check_dtype_enum.sh new file mode 100755 index 00000000..3f957a47 --- /dev/null +++ b/fortran/test/check_dtype_enum.sh @@ -0,0 +1,96 @@ +#!/usr/bin/env bash +# (C) Copyright 2026- ECMWF and individual contributors. +# +# This software is licensed under the terms of the Apache Licence Version 2.0 +# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +# In applying this licence, ECMWF does not waive the privileges and immunities +# granted to it by virtue of its status as an intergovernmental organisation nor +# does it submit to any jurisdiction. +# +# Single source of truth for element types (AGENTS.md): the C `tgm_dtype` enum +# in the checked-in tensogram.h must cover exactly the variants of the Rust core +# `Dtype` in rust/tensogram/src/dtype.rs — no more, no fewer. +# +# The Rust side already fails to compile when a core variant is added (the +# `From for TgmDtype` conversion and its unit test both match +# exhaustively). This guard covers what a Rust test cannot: tensogram.h is a +# *checked-in generated artefact* that the C++ and Fortran bindings mirror by +# hand, so a stale or hand-edited header can disagree with the crate it was +# supposed to be generated from. +# +# Deliberately compares NAME SETS, not name=value pairs, unlike +# check_frame_type_enum.sh. Frame-type numbers are wire-format values and must +# mirror the core discriminants; dtype numbers are not on the wire (the format +# stores dtype as text) — they are frozen C ABI values that must NOT follow a +# reordering of the core enum. What the guard does check about the numbers is +# that they stay a dense, duplicate-free 0..N-1 range, which is the property +# the bindings' lookup tables rely on. +# +# Usage: check_dtype_enum.sh + +set -euo pipefail + +DTYPE_RS="$1" +HDR="$2" + +# Header: enum members appear one per line as ` TGM_DTYPE_NAME = ,` inside +# the `typedef enum { ... } tgm_dtype;` block. Doc-comment references start +# with ` * ` so they never match `^[[:space:]]*TGM_...`. +extract_hdr_pairs() { + grep -oE '^[[:space:]]*TGM_DTYPE_[A-Z0-9_]+[[:space:]]*=[[:space:]]*[0-9]+' "$1" | + grep -oE 'TGM_DTYPE_[A-Z0-9_]+[[:space:]]*=[[:space:]]*[0-9]+' | + sed -E 's/[[:space:]]*=[[:space:]]*/=/' | + sort -u +} + +extract_hdr_names() { + extract_hdr_pairs "$1" | cut -d= -f1 | sort -u +} + +# Rust core: the bare `Variant,` lines of `pub enum Dtype { ... }`. Digits are +# part of the name (`Float16` -> `FLOAT16`), so underscores are inserted only at +# lower->upper boundaries — the same spelling cbindgen produces. +extract_core_names() { + sed -n '/^pub enum Dtype {/,/^}/p' "$1" | + grep -oE '^[[:space:]]+[A-Za-z][A-Za-z0-9]*,' | + tr -d ' ,' | + sed -E 's/([a-z0-9])([A-Z])/\1_\2/g' | + tr '[:lower:]' '[:upper:]' | + sed -E 's/^/TGM_DTYPE_/' | + sort -u +} + +# `|| true` so a missing file or a moved declaration reaches the explicit +# diagnostic below instead of dying on a bare non-zero from the pipeline. +hdr_names=$(extract_hdr_names "$HDR") || true +core_names=$(extract_core_names "$DTYPE_RS") || true + +# A vacuous pass (both sides empty because a path or a declaration moved) is +# worse than a failure — it looks green while guarding nothing. +if [ -z "$hdr_names" ] || [ -z "$core_names" ]; then + echo "dtype enum guard extracted no variants — check both paths exist" + echo " C header : $HDR" + echo " Rust core : $DTYPE_RS" + exit 1 +fi + +if [ "$hdr_names" != "$core_names" ]; then + echo "dtype enum mismatch (C header vs Rust core Dtype):" + echo " C header : $HDR" + echo " Rust core : $DTYPE_RS" + diff <(printf '%s\n' "$hdr_names") <(printf '%s\n' "$core_names") || true + exit 1 +fi + +# The C values must stay a dense 0..N-1 range with no duplicates: the bindings +# index lookup tables by them. +n=$(printf '%s\n' "$hdr_names" | grep -c .) +values=$(extract_hdr_pairs "$HDR" | cut -d= -f2 | sort -n) +expected=$(seq 0 $((n - 1))) +if [ "$values" != "$expected" ]; then + echo "dtype enum values are not a dense 0..$((n - 1)) range:" + printf '%s\n' "$(extract_hdr_pairs "$HDR")" + exit 1 +fi + +echo "dtype enum consistency OK (${n} dtypes match, values 0..$((n - 1)))" diff --git a/fortran/test/check_enum_mirror.sh b/fortran/test/check_enum_mirror.sh new file mode 100755 index 00000000..3c485fab --- /dev/null +++ b/fortran/test/check_enum_mirror.sh @@ -0,0 +1,86 @@ +#!/usr/bin/env bash +# (C) Copyright 2026- ECMWF and individual contributors. +# +# This software is licensed under the terms of the Apache Licence Version 2.0 +# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +# In applying this licence, ECMWF does not waive the privileges and immunities +# granted to it by virtue of its status as an intergovernmental organisation nor +# does it submit to any jurisdiction. +# +# Single source of truth for the VALUED C enums the Fortran binding mirrors +# (AGENTS.md): a `TGM__*` integer parameter block in tensogram.F90 must +# match the `typedef enum { ... }` of the same name in tensogram.h exactly — +# same member names, same numbers, no extras on either side. Fails (non-zero) +# on any drift, so a variant added to (or renumbered in) the header without +# updating the Fortran mirror is caught at CI time. +# +# Generalises check_error_enum.sh / check_value_type_enum.sh, which do the same +# job for one enum each, to every enum whose header members carry an explicit +# `= `: TGM_FRAME_TYPE (wire-format frame-type codes), +# TGM_DTYPE / TGM_BYTE_ORDER (frozen C ABI codes for the typed object +# accessors), TGM_AGGREGATE_HASH_POLICY and TGM_COMPRESSION_BACKEND (the +# encode-option knobs). One script, one test per enum — the extraction is +# identical, only the prefix differs. +# +# Note what this does NOT need to re-check. The name=value comparison +# preserves structural quirks by construction: the frame-type sequence's +# deliberate GAP AT 4 (the obsolete v2 data-object layout, reserved and +# undeclared) cannot appear on the Fortran side unless it appears in the +# header, which check_frame_type_enum.sh separately forbids; and the dtype +# codes' dense 0..N-1 range is guarded against the Rust core by +# check_dtype_enum.sh. This guard covers the remaining edge those two do not: +# a hand-maintained Fortran mirror drifting from the generated header. +# +# Usage: check_enum_mirror.sh + +set -euo pipefail + +F90="$1" +HDR="$2" +PREFIX="$3" + +# Header: enum members appear one per line as ` TGM__NAME = ,` inside +# the `typedef enum { ... }` block. Doc-comment references start with ` * ` so +# they never match `^[[:space:]]*TGM_...`, and they carry no `= ` anyway. +extract_hdr() { + grep -oE "^[[:space:]]*${PREFIX}_[A-Z0-9_]+[[:space:]]*=[[:space:]]*[0-9]+" "$HDR" | + grep -oE "${PREFIX}_[A-Z0-9_]+[[:space:]]*=[[:space:]]*[0-9]+" | + sed -E 's/[[:space:]]*=[[:space:]]*/=/' | + sort -u +} + +# Fortran: `integer(c_int), parameter :: TGM__NAME = ` declarations. +# Anchoring on `parameter ::` keeps prose in the doc comments (which name the +# parameters freely) from being mistaken for a declaration. +extract_f90() { + grep -E 'parameter[[:space:]]*::' "$F90" | + grep -oE "${PREFIX}_[A-Z0-9_]+[[:space:]]*=[[:space:]]*[0-9]+" | + sed -E 's/[[:space:]]*=[[:space:]]*/=/' | + sort -u +} + +# `|| true` so a missing file or a moved declaration reaches the explicit +# diagnostic below instead of dying on a bare non-zero from the pipeline. +hdr=$(extract_hdr) || true +f90=$(extract_f90) || true + +# A vacuous pass (both sides empty because a path, a prefix or a declaration +# moved) is worse than a failure — it looks green while guarding nothing. +if [ -z "$hdr" ] || [ -z "$f90" ]; then + echo "${PREFIX} enum guard extracted no variants — check the paths and the prefix" + echo " C header : $HDR" + echo " Fortran : $F90" + echo " prefix : $PREFIX" + exit 1 +fi + +if [ "$hdr" != "$f90" ]; then + echo "${PREFIX} enum mismatch (Fortran mirror vs C header):" + echo " C header : $HDR" + echo " Fortran : $F90" + diff <(printf '%s\n' "$hdr") <(printf '%s\n' "$f90") || true + exit 1 +fi + +n=$(printf '%s\n' "$hdr" | grep -c .) +echo "${PREFIX} enum consistency OK (${n} members match)" diff --git a/fortran/test/check_frame_type_enum.sh b/fortran/test/check_frame_type_enum.sh new file mode 100755 index 00000000..624af9e1 --- /dev/null +++ b/fortran/test/check_frame_type_enum.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# (C) Copyright 2026- ECMWF and individual contributors. +# +# This software is licensed under the terms of the Apache Licence Version 2.0 +# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +# In applying this licence, ECMWF does not waive the privileges and immunities +# granted to it by virtue of its status as an intergovernmental organisation nor +# does it submit to any jurisdiction. +# +# Single source of truth for frame-type codes (AGENTS.md): the C +# `tgm_frame_type` enum in tensogram.h must mirror the Rust core `FrameType` +# discriminants in rust/tensogram/src/wire.rs exactly. Those numbers ARE the +# wire format's frame-type field (plans/WIRE_FORMAT.md §2.2), so any skew +# silently mislabels frames in every C-family binding rather than failing +# loudly. Fails (non-zero) on any drift. +# +# Note the deliberate gap at 4 — the obsolete v2 data-object layout, reserved +# and unusable, so neither side declares it. The guard compares name=value +# pairs, so the gap is preserved by construction and re-checked explicitly. +# +# Usage: check_frame_type_enum.sh + +set -euo pipefail + +WIRE="$1" +HDR="$2" + +# Header: enum members appear one per line as ` TGM_FRAME_TYPE_NAME = ,` +# inside the `typedef enum { ... } tgm_frame_type;` block. Doc-comment +# references start with ` * ` so they never match `^[[:space:]]*TGM_...`. +extract_hdr() { + grep -oE '^[[:space:]]*TGM_FRAME_TYPE_[A-Z_]+[[:space:]]*=[[:space:]]*[0-9]+' "$1" | + grep -oE 'TGM_FRAME_TYPE_[A-Z_]+[[:space:]]*=[[:space:]]*[0-9]+' | + sed -E 's/[[:space:]]*=[[:space:]]*/=/' | + sort -u +} + +# Rust variant name -> the C spelling cbindgen produces for the mirrored FFI +# enum: drop the redundant `Frame` suffix (`NTensorFrame` -> `NTensor`), then +# upper-case with `_` inserted at each lower->upper boundary +# (`NTensor` -> `NTENSOR`, `HeaderMetadata` -> `HEADER_METADATA`). +camel_to_screaming() { + printf '%s' "${1%Frame}" | + sed -E 's/([a-z0-9])([A-Z])/\1_\2/g' | + tr '[:lower:]' '[:upper:]' +} + +# Rust core: the `Variant = ,` lines of `pub enum FrameType { ... }`. +# Comment lines (`// Type 4 reserved …`, `/// …`) carry no `=` digit pair and +# are skipped. +extract_wire() { + sed -n '/^pub enum FrameType {/,/^}/p' "$1" | + grep -oE '^[[:space:]]+[A-Za-z0-9]+[[:space:]]*=[[:space:]]*[0-9]+' | + sed -E 's/[[:space:]]//g' | + while IFS='=' read -r name value; do + printf 'TGM_FRAME_TYPE_%s=%s\n' "$(camel_to_screaming "$name")" "$value" + done | + sort -u +} + +# `|| true` so a missing file or a moved declaration reaches the explicit +# diagnostic below instead of dying on a bare non-zero from the pipeline. +hdr=$(extract_hdr "$HDR") || true +wire=$(extract_wire "$WIRE") || true + +# A vacuous pass (both sides empty because a path or a declaration moved) is +# worse than a failure — it looks green while guarding nothing. +if [ -z "$hdr" ] || [ -z "$wire" ]; then + echo "frame-type enum guard extracted no variants — check both paths exist" + echo " C header : $HDR" + echo " Rust core : $WIRE" + exit 1 +fi + +if [ "$hdr" != "$wire" ]; then + echo "frame-type enum mismatch:" + echo " C header : $HDR" + echo " Rust core : $WIRE" + diff <(printf '%s\n' "$hdr") <(printf '%s\n' "$wire") || true + exit 1 +fi + +# Type 4 is reserved by the wire format and must never reappear. +if printf '%s\n' "$hdr" | grep -qE '=4$'; then + echo "frame type 4 is reserved (obsolete v2 data-object layout) but is declared:" + printf '%s\n' "$hdr" | grep -E '=4$' + exit 1 +fi + +n=$(printf '%s\n' "$hdr" | grep -c .) +echo "frame-type enum consistency OK (${n} types match)" diff --git a/fortran/test/test_encode_options.f90 b/fortran/test/test_encode_options.f90 new file mode 100644 index 00000000..cd0565e2 --- /dev/null +++ b/fortran/test/test_encode_options.f90 @@ -0,0 +1,444 @@ +! (C) Copyright 2026- ECMWF and individual contributors. +! +! This software is licensed under the terms of the Apache Licence Version 2.0 +! which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +! In applying this licence, ECMWF does not waive the privileges and immunities +! granted to it by virtue of its status as an intergovernmental organisation nor +! does it submit to any jurisdiction. + +!> Deferred Wave-B, component C — the full encode-side option set +!> (TgmEncodeOptions), which supersedes the mask-only options by also carrying +!> the hash algorithm, the aggregate-hash PLACEMENT and the codec BACKEND: +!> * tensogram_encode_with_options (tgm_encode_with_encode_options) +!> * tensogram_file_append_with_options (tgm_file_append_with_encode_options) +!> * tensogram_streaming_encoder_create_with_options (tgm_streaming_encoder_create_with_encode_options) +!> +!> The aggregate-hash assertions are a CLOSED LOOP: what the option asked for +!> is verified by walking the resulting message with this binding's own frame +!> walker (component A3) and by reading its message header. +program test_encode_options + use, intrinsic :: iso_c_binding + use, intrinsic :: ieee_arithmetic + use tensogram + implicit none + + integer, parameter :: MAXF = 32 + integer :: npass + + npass = 0 + + call default_options_match_plain_encode() + call aggregate_hash_placements() + call aggregate_hash_both_is_visible_in_the_frames() + call hash_algorithm_knob() + call compression_backend_knob() + call mask_options_still_reachable() + call file_append_with_options_cases() + call streaming_create_with_options_cases() + + print '(a,i0,a)', 'test_encode_options: PASS (', npass, ' checks)' + +contains + + ! ---- No options at all == the library defaults -------------------------- + subroutine default_options_match_plain_encode() + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_frame) :: frames(MAXF) + integer(c_int) :: err + integer :: n + call encode_opts(wire, err) + call assert(err == TGM_ERROR_OK, 'defaults: encode OK') + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'defaults: walk OK') + ! Buffered + default hashing => the aggregate hash frame goes in the header. + call assert(seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), 'defaults: header hash frame') + call assert(.not. seen(frames, n, TGM_FRAME_TYPE_FOOTER_HASH), 'defaults: no footer hash frame') + call assert(values_round_trip(wire), 'defaults: values round-trip') + end subroutine default_options_match_plain_encode + + ! ---- AUTO / NONE / HEADER / FOOTER placements --------------------------- + subroutine aggregate_hash_placements() + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_frame) :: frames(MAXF) + integer(c_int) :: err + integer :: n, k + + call encode_opts(wire, err, aggregate_hash=TGM_AGGREGATE_HASH_POLICY_AUTO) + call assert(err == TGM_ERROR_OK, 'AUTO: encode OK') + call walk(wire, frames, n, err) + call assert(seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), 'AUTO: buffered => header hash') + call assert(.not. seen(frames, n, TGM_FRAME_TYPE_FOOTER_HASH), 'AUTO: no footer hash') + + call encode_opts(wire, err, aggregate_hash=TGM_AGGREGATE_HASH_POLICY_NONE) + call assert(err == TGM_ERROR_OK, 'NONE: encode OK') + call walk(wire, frames, n, err) + call assert(.not. seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), 'NONE: no header hash frame') + call assert(.not. seen(frames, n, TGM_FRAME_TYPE_FOOTER_HASH), 'NONE: no footer hash frame') + ! NONE drops only the AGGREGATE frame — the per-frame inline slots stay. + do k = 1, n + call assert(frames(k)%has_hash(), 'NONE: per-frame inline hashes are unaffected') + end do + + call encode_opts(wire, err, aggregate_hash=TGM_AGGREGATE_HASH_POLICY_HEADER) + call assert(err == TGM_ERROR_OK, 'HEADER: encode OK') + call walk(wire, frames, n, err) + call assert(seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), 'HEADER: header hash frame') + call assert(.not. seen(frames, n, TGM_FRAME_TYPE_FOOTER_HASH), 'HEADER: no footer hash frame') + + call encode_opts(wire, err, aggregate_hash=TGM_AGGREGATE_HASH_POLICY_FOOTER) + call assert(err == TGM_ERROR_OK, 'FOOTER: encode OK') + call walk(wire, frames, n, err) + call assert(.not. seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), 'FOOTER: no header hash frame') + call assert(seen(frames, n, TGM_FRAME_TYPE_FOOTER_HASH), 'FOOTER: footer hash frame') + call assert(values_round_trip(wire), 'FOOTER: values round-trip') + end subroutine aggregate_hash_placements + + ! ---- BOTH: a hash frame in the header AND in the footer ----------------- + subroutine aggregate_hash_both_is_visible_in_the_frames() + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_frame) :: frames(MAXF) + type(tensogram_message_header) :: hdr + integer(c_int) :: err + integer :: n, k, header_hashes, footer_hashes + + call encode_opts(wire, err, aggregate_hash=TGM_AGGREGATE_HASH_POLICY_BOTH) + call assert(err == TGM_ERROR_OK, 'BOTH: encode OK') + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'BOTH: walk OK') + + header_hashes = 0 + footer_hashes = 0 + do k = 1, n + if (frames(k)%frame_type() == TGM_FRAME_TYPE_HEADER_HASH) header_hashes = header_hashes + 1 + if (frames(k)%frame_type() == TGM_FRAME_TYPE_FOOTER_HASH) footer_hashes = footer_hashes + 1 + end do + call assert(header_hashes == 1, 'BOTH: exactly one header hash frame') + call assert(footer_hashes == 1, 'BOTH: exactly one footer hash frame') + ! Both frames carry the same hash list, so their payloads are identical. + call assert(bytes_eq(payload_of(frames, n, TGM_FRAME_TYPE_HEADER_HASH), & + payload_of(frames, n, TGM_FRAME_TYPE_FOOTER_HASH)), & + 'BOTH: header and footer hash frames carry identical lists') + ! The header frame precedes the data objects; the footer frame follows them. + call assert(index_of(frames, n, TGM_FRAME_TYPE_HEADER_HASH) < & + index_of(frames, n, TGM_FRAME_TYPE_NTENSOR), 'BOTH: header hash comes first') + call assert(index_of(frames, n, TGM_FRAME_TYPE_FOOTER_HASH) > & + index_of(frames, n, TGM_FRAME_TYPE_NTENSOR), 'BOTH: footer hash comes last') + + ! The preamble advertises both, matching the frames exactly. + call tensogram_message_header_read(wire, hdr, err) + call assert(err == TGM_ERROR_OK, 'BOTH: header read OK') + call assert(hdr%has_header_hashes(), 'BOTH: preamble flags header hashes') + call assert(hdr%has_footer_hashes(), 'BOTH: preamble flags footer hashes') + call assert(values_round_trip(wire), 'BOTH: values round-trip') + end subroutine aggregate_hash_both_is_visible_in_the_frames + + ! ---- The hash algorithm now lives in the options ------------------------ + subroutine hash_algorithm_knob() + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_frame) :: frames(MAXF) + integer(c_int) :: err + integer :: n, k + + ! '' disables hashing entirely: no aggregate frame, no inline slots. + call encode_opts(wire, err, hash='') + call assert(err == TGM_ERROR_OK, 'hash="": encode OK') + call walk(wire, frames, n, err) + call assert(.not. seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), 'hash="": no aggregate frame') + do k = 1, n + call assert(.not. frames(k)%has_hash(), 'hash="": no per-frame inline hash') + end do + + call encode_opts(wire, err, hash='xxh3') + call assert(err == TGM_ERROR_OK, 'hash="xxh3": encode OK') + call walk(wire, frames, n, err) + call assert(seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), 'hash="xxh3": aggregate frame') + + call encode_opts(wire, err, hash='not-an-algorithm') + call assert(err == TGM_ERROR_INVALID_ARG, 'unknown hash algorithm -> INVALID_ARG') + end subroutine hash_algorithm_knob + + ! ---- The codec backend is an implementation choice, not a format one --- + subroutine compression_backend_knob() + integer(c_int8_t), allocatable :: wire_auto(:), wire_ffi(:), wire_pure(:) + integer(c_int) :: err + call encode_opts(wire_auto, err, compression='zstd', & + compression_backend=TGM_COMPRESSION_BACKEND_AUTO) + call assert(err == TGM_ERROR_OK, 'backend AUTO: encode OK') + call encode_opts(wire_ffi, err, compression='zstd', & + compression_backend=TGM_COMPRESSION_BACKEND_FFI) + call assert(err == TGM_ERROR_OK, 'backend FFI: encode OK') + call encode_opts(wire_pure, err, compression='zstd', & + compression_backend=TGM_COMPRESSION_BACKEND_PURE) + call assert(err == TGM_ERROR_OK, 'backend PURE: encode OK') + ! Whichever codec produced the bytes, the values come back bit-identical. + call assert(values_round_trip(wire_auto), 'backend AUTO: lossless round-trip') + call assert(values_round_trip(wire_ffi), 'backend FFI: lossless round-trip') + call assert(values_round_trip(wire_pure), 'backend PURE: lossless round-trip') + end subroutine compression_backend_knob + + ! ---- The superseded mask knobs are still reachable ---------------------- + subroutine mask_options_still_reachable() + real(c_float) :: vals(4) + integer(c_int8_t), allocatable :: data(:), wire(:), mask(:) + integer(c_size_t) :: lens(1) + type(tensogram_buffer) :: buf + type(tensogram_message) :: msg + integer(c_int) :: err + vals(1) = 1.0_c_float + vals(2) = ieee_value(1.0_c_float, ieee_quiet_nan) + vals(3) = ieee_value(1.0_c_float, ieee_positive_inf) + vals(4) = 2.0_c_float + data = transfer(vals, [0_c_int8_t], 4 * 4) + lens(1) = int(4 * 4, c_size_t) + + ! Without allow_nan the encoder rejects the non-finite input … + call tensogram_encode_with_options(json1(4, 'none'), data, lens, buf, err) + call assert(err /= TGM_ERROR_OK, 'masks: non-finite rejected by default') + + ! … and with it the companion masks are written and recoverable. + call tensogram_encode_with_options(json1(4, 'none'), data, lens, buf, err, & + allow_nan=.true., allow_inf=.true., & + nan_mask_method='rle', & + small_mask_threshold_bytes=0_c_int64_t) + call assert(err == TGM_ERROR_OK, 'masks: allow_nan/allow_inf + explicit method') + call buf%as_array(wire) + call tensogram_decode_with_masks(wire, msg, err) + call assert(err == TGM_ERROR_OK, 'masks: decode_with_masks OK') + call assert(tensogram_object_has_masks(msg, 1), 'masks: object carries masks') + call tensogram_object_mask(msg, 1, TGM_MASK_KIND_NAN, mask, err) + call assert(err == TGM_ERROR_OK .and. size(mask) == 4, 'masks: NaN mask has one byte per element') + call assert(mask(2) == 1_c_int8_t, 'masks: NaN position recorded') + call tensogram_object_mask(msg, 1, TGM_MASK_KIND_POS_INF, mask, err) + call assert(err == TGM_ERROR_OK .and. mask(3) == 1_c_int8_t, 'masks: +Inf position recorded') + end subroutine mask_options_still_reachable + + ! ---- file_append_with_options ------------------------------------------- + subroutine file_append_with_options_cases() + character(len=*), parameter :: path = 'test_encode_options_append.tgm' + real(c_float) :: vals(4) + integer(c_int8_t), allocatable :: data(:), wire(:) + integer(c_size_t) :: lens(1) + type(tensogram_file) :: f + type(tensogram_buffer) :: raw + type(tensogram_frame) :: frames(MAXF) + integer(c_int) :: err + integer :: n, unit, ios + vals = [1.0_c_float, 2.0_c_float, 3.0_c_float, 4.0_c_float] + data = transfer(vals, [0_c_int8_t], 4 * 4) + lens(1) = int(4 * 4, c_size_t) + + call tensogram_file_create(path, f, err) + call assert(err == TGM_ERROR_OK, 'append_with_options: create') + call tensogram_file_append_with_options(f, json1(4, 'none'), data, lens, err, & + aggregate_hash=TGM_AGGREGATE_HASH_POLICY_BOTH) + call assert(err == TGM_ERROR_OK, 'append_with_options: append BOTH') + call tensogram_file_append_with_options(f, json1(4, 'zstd'), data, lens, err, & + compression_backend=TGM_COMPRESSION_BACKEND_PURE) + call assert(err == TGM_ERROR_OK, 'append_with_options: append zstd via the pure backend') + call f%close() + + call tensogram_file_open(path, f, err) + call assert(err == TGM_ERROR_OK, 'append_with_options: reopen') + call tensogram_file_read_message(f, 1, raw, err) + call assert(err == TGM_ERROR_OK, 'append_with_options: read message 1') + call raw%as_array(wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'append_with_options: walk message 1') + call assert(seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), 'append_with_options: header hash frame') + call assert(seen(frames, n, TGM_FRAME_TYPE_FOOTER_HASH), 'append_with_options: footer hash frame') + call f%close() + + open(newunit=unit, file=path, status='old', iostat=ios) + if (ios == 0) close(unit, status='delete') + end subroutine file_append_with_options_cases + + ! ---- streaming_encoder_create_with_options ------------------------------ + subroutine streaming_create_with_options_cases() + character(len=*), parameter :: path = 'test_encode_options_stream.tgm' + real(c_float) :: vals(4) + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_streaming_encoder) :: enc + type(tensogram_file) :: f + type(tensogram_buffer) :: raw + type(tensogram_frame) :: frames(MAXF) + integer(c_int) :: err + integer :: n, unit, ios + vals = [1.0_c_float, 2.0_c_float, 3.0_c_float, 4.0_c_float] + + ! HEADER / BOTH cannot work when the header is written first. + call tensogram_streaming_encoder_create_with_options(path, enc, err, & + aggregate_hash=TGM_AGGREGATE_HASH_POLICY_HEADER) + call assert(err == TGM_ERROR_ENCODING, 'streaming: HEADER policy rejected') + call tensogram_streaming_encoder_create_with_options(path, enc, err, & + aggregate_hash=TGM_AGGREGATE_HASH_POLICY_BOTH) + call assert(err == TGM_ERROR_ENCODING, 'streaming: BOTH policy rejected') + call assert(len(tensogram_last_error()) > 0, 'streaming: rejection explains itself') + + ! FOOTER (and AUTO, which resolves to it) is the valid streaming choice. + call tensogram_streaming_encoder_create_with_options(path, enc, err, & + aggregate_hash=TGM_AGGREGATE_HASH_POLICY_FOOTER, & + compression_backend=TGM_COMPRESSION_BACKEND_PURE) + call assert(err == TGM_ERROR_OK, 'streaming: FOOTER policy accepted') + call tensogram_streaming_encoder_write(enc, vals, err, compression='zstd') + call assert(err == TGM_ERROR_OK, 'streaming: write object') + call tensogram_streaming_encoder_finish(enc, err) + call assert(err == TGM_ERROR_OK, 'streaming: finish') + call enc%free() + + call tensogram_file_open(path, f, err) + call assert(err == TGM_ERROR_OK, 'streaming: reopen') + call tensogram_file_read_message(f, 1, raw, err) + call assert(err == TGM_ERROR_OK, 'streaming: read message') + call raw%as_array(wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'streaming: walk') + call assert(seen(frames, n, TGM_FRAME_TYPE_FOOTER_HASH), 'streaming: footer hash frame') + call assert(.not. seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), 'streaming: no header hash frame') + call f%close() + + open(newunit=unit, file=path, status='old', iostat=ios) + if (ios == 0) close(unit, status='delete') + end subroutine streaming_create_with_options_cases + + ! ---- helpers ------------------------------------------------------------ + + !> Encode the fixed 4-element float32 fixture through + !> tensogram_encode_with_options, forwarding whichever knobs were given. + subroutine encode_opts(wire, err, hash, aggregate_hash, compression_backend, compression) + integer(c_int8_t), allocatable, intent(out) :: wire(:) + integer(c_int), intent(out) :: err + character(len=*), intent(in), optional :: hash, compression + integer(c_int), intent(in), optional :: aggregate_hash, compression_backend + real(c_float) :: vals(4) + integer(c_int8_t), allocatable :: data(:) + integer(c_size_t) :: lens(1) + character(len=:), allocatable :: cmp + type(tensogram_buffer) :: buf + vals = [1.0_c_float, 2.0_c_float, 3.0_c_float, 4.0_c_float] + data = transfer(vals, [0_c_int8_t], 4 * 4) + lens(1) = int(4 * 4, c_size_t) + cmp = 'none' + if (present(compression)) cmp = compression + call tensogram_encode_with_options(json1(4, cmp), data, lens, buf, err, & + hash=hash, aggregate_hash=aggregate_hash, & + compression_backend=compression_backend) + if (err == TGM_ERROR_OK) then + call buf%as_array(wire) + else + allocate(wire(0)) + end if + end subroutine encode_opts + + !> Decode `wire` and check the fixture's values survived bit-exactly. + logical function values_round_trip(wire) + integer(c_int8_t), intent(in) :: wire(:) + real(c_float), allocatable :: got(:) + type(tensogram_message) :: msg + integer(c_int) :: err + values_round_trip = .false. + call tensogram_decode(wire, msg, err) + if (err /= TGM_ERROR_OK) return + call tensogram_to_array(msg, 1, got, err) + if (err /= TGM_ERROR_OK) return + if (size(got) /= 4) return + values_round_trip = all(transfer(got, [0_c_int32_t]) == & + transfer([1.0_c_float, 2.0_c_float, 3.0_c_float, 4.0_c_float], & + [0_c_int32_t])) + end function values_round_trip + + !> One-object descriptor envelope: `n` float32 elements, pipeline `cmp`. + function json1(n, cmp) result(s) + integer, intent(in) :: n + character(len=*), intent(in) :: cmp + character(len=:), allocatable :: s + character(len=32) :: ns + write (ns, '(i0)') n + s = '{"descriptors":[{"type":"ndarray","ndim":1,"shape":[' // trim(ns) // & + '],"strides":[1],"dtype":"float32","byte_order":"' // host_bo() // & + '","encoding":"none","filter":"none","compression":"' // cmp // '"}]}' + end function json1 + + !> Walk `wire` to exhaustion, collecting the frames (component A3). + subroutine walk(wire, frames, n, err) + integer(c_int8_t), intent(in) :: wire(:) + type(tensogram_frame), intent(out) :: frames(:) + integer, intent(out) :: n + integer(c_int), intent(out) :: err + type(tensogram_frame_iterator) :: it + type(tensogram_frame) :: fr + logical :: found + n = 0 + call tensogram_frames(wire, it, err) + if (err /= TGM_ERROR_OK) return + do + call it%next(fr, found, err) + if (.not. found) exit + n = n + 1 + call assert(n <= size(frames), 'walk: frame buffer overflow') + frames(n) = fr + end do + call it%free() + end subroutine walk + + logical function seen(frames, n, ftype) + type(tensogram_frame), intent(in) :: frames(:) + integer, intent(in) :: n + integer(c_int), intent(in) :: ftype + seen = index_of(frames, n, ftype) > 0 + end function seen + + !> Position of the first frame of type `ftype` (0 when absent). + integer function index_of(frames, n, ftype) + type(tensogram_frame), intent(in) :: frames(:) + integer, intent(in) :: n + integer(c_int), intent(in) :: ftype + integer :: k + index_of = 0 + do k = n, 1, -1 + if (frames(k)%frame_type() == ftype) index_of = k + end do + end function index_of + + !> Payload bytes of the first frame of type `ftype` (size 0 when absent). + function payload_of(frames, n, ftype) result(p) + type(tensogram_frame), intent(in) :: frames(:) + integer, intent(in) :: n + integer(c_int), intent(in) :: ftype + integer(c_int8_t), allocatable :: p(:) + integer :: k + k = index_of(frames, n, ftype) + if (k == 0) then + allocate(p(0)) + else + p = frames(k)%payload() + end if + end function payload_of + + function host_bo() result(bo) + character(len=:), allocatable :: bo + integer(c_int8_t) :: probe(4) + probe = transfer(1_c_int32_t, 0_c_int8_t, 4) + if (probe(1) == 1_c_int8_t) then + bo = 'little' + else + bo = 'big' + end if + end function host_bo + + logical function bytes_eq(x, y) + integer(c_int8_t), intent(in) :: x(:), y(:) + bytes_eq = size(x) == size(y) .and. size(x) > 0 + if (bytes_eq) bytes_eq = all(x == y) + end function bytes_eq + + subroutine assert(cond, what) + logical, intent(in) :: cond + character(len=*), intent(in) :: what + if (.not. cond) then + print '(a,a)', 'test_encode_options: FAIL: ', what + error stop 1 + end if + npass = npass + 1 + end subroutine assert + +end program test_encode_options diff --git a/fortran/test/test_frames.f90 b/fortran/test/test_frames.f90 new file mode 100644 index 00000000..f499a94d --- /dev/null +++ b/fortran/test/test_frames.f90 @@ -0,0 +1,546 @@ +! (C) Copyright 2026- ECMWF and individual contributors. +! +! This software is licensed under the terms of the Apache Licence Version 2.0 +! which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +! In applying this licence, ECMWF does not waive the privileges and immunities +! granted to it by virtue of its status as an intergovernmental organisation nor +! does it submit to any jurisdiction. + +!> Deferred Wave-B, component A3 — the lazy frame walker and the typed message +!> header: +!> * TGM_FRAME_TYPE_* (tgm_frame_type, note the reserved gap at 4) +!> * tensogram_frame (TgmFrame, as a copyable VALUE type) +!> * tensogram_frame_iterator (tgm_frame_iter_create / _next / _free) +!> * tensogram_message_header_read (tgm_message_header) +!> +!> The structural assertions are statements about the wire format +!> (plans/WIRE_FORMAT.md): a 24-byte preamble, 16-byte frame headers, a 20-byte +!> footer on NTENSOR frames and 12 bytes on every other type, frames that tile +!> the message without overlapping, and a 24-byte postamble that is NOT a frame. +program test_frames + use, intrinsic :: iso_c_binding + use tensogram + implicit none + + ! Wire-format structural constants (plans/WIRE_FORMAT.md §§2, 3, 7). + integer, parameter :: PREAMBLE = 24 + integer, parameter :: POSTAMBLE = 24 + integer, parameter :: FRAME_HEADER = 16 + integer, parameter :: FOOTER_ANY = 12 ! [hash][ENDF] + integer, parameter :: FOOTER_NTEN = 20 ! [cbor_offset][hash][ENDF] + integer, parameter :: HASH_PRESENT = 2 ! §2.5 frame flag bit 1 + integer, parameter :: MAXF = 32 ! plenty for these fixtures + + integer :: npass + + npass = 0 + + call frame_type_constants() + call default_values() + call buffered_frame_sequence() + call frame_geometry() + call payload_and_hash_flags() + call payload_outlives_the_iterator() + call streaming_frame_sequence() + call end_versus_malformed() + call bad_buffers_rejected() + call message_header_buffered() + call message_header_streaming() + + print '(a,i0,a)', 'test_frames: PASS (', npass, ' checks)' + +contains + + ! ---- The mirrored wire numbers, gap at 4 included ----------------------- + subroutine frame_type_constants() + call assert(TGM_FRAME_TYPE_HEADER_METADATA == 1, 'frame type HEADER_METADATA = 1') + call assert(TGM_FRAME_TYPE_HEADER_INDEX == 2, 'frame type HEADER_INDEX = 2') + call assert(TGM_FRAME_TYPE_HEADER_HASH == 3, 'frame type HEADER_HASH = 3') + ! 4 is reserved (the obsolete v2 data-object layout) — no parameter. + call assert(TGM_FRAME_TYPE_FOOTER_HASH == 5, 'frame type FOOTER_HASH = 5') + call assert(TGM_FRAME_TYPE_FOOTER_INDEX == 6, 'frame type FOOTER_INDEX = 6') + call assert(TGM_FRAME_TYPE_FOOTER_METADATA == 7, 'frame type FOOTER_METADATA = 7') + call assert(TGM_FRAME_TYPE_PRECEDER_METADATA == 8, 'frame type PRECEDER_METADATA = 8') + call assert(TGM_FRAME_TYPE_NTENSOR == 9, 'frame type NTENSOR = 9') + end subroutine frame_type_constants + + ! ---- A default-initialised frame / header is inert, never a trap -------- + subroutine default_values() + type(tensogram_frame) :: fr + type(tensogram_message_header) :: hdr + call assert(fr%frame_type() == 0, 'default frame: no type') + call assert(fr%version() == 0 .and. fr%flags() == 0, 'default frame: no version / flags') + call assert(fr%offset() == 0_c_size_t .and. fr%length() == 0_c_size_t, 'default frame: no span') + call assert(.not. fr%has_hash(), 'default frame: no inline hash') + ! %payload() must answer with an empty array rather than an unallocated + ! one — the component is only allocated by a successful %next. + call assert(size(fr%payload()) == 0, 'default frame: empty payload') + call assert(hdr%version() == 0, 'default header: no version') + call assert(hdr%total_length() == 0_c_int64_t, 'default header: no length') + call assert(.not. hdr%has_header_metadata() .and. .not. hdr%has_footer_metadata() .and. & + .not. hdr%has_header_index() .and. .not. hdr%has_footer_index() .and. & + .not. hdr%has_header_hashes() .and. .not. hdr%has_footer_hashes() .and. & + .not. hdr%has_preceder_metadata() .and. .not. hdr%has_hashes_present(), & + 'default header: every flag clear') + end subroutine default_values + + ! ---- The expected frame-type sequence of a buffered message ------------- + subroutine buffered_frame_sequence() + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_frame) :: frames(MAXF) + integer(c_int) :: err + integer :: n + + call buffered(2, .true., wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'buffered walk: clean end') + call assert(n == 5, 'buffered 2-object message: 5 frames') + call assert(frames(1)%frame_type() == TGM_FRAME_TYPE_HEADER_METADATA, 'buffered[1] header metadata') + call assert(frames(2)%frame_type() == TGM_FRAME_TYPE_HEADER_INDEX, 'buffered[2] header index') + call assert(frames(3)%frame_type() == TGM_FRAME_TYPE_HEADER_HASH, 'buffered[3] header hash') + call assert(frames(4)%frame_type() == TGM_FRAME_TYPE_NTENSOR, 'buffered[4] ntensor') + call assert(frames(5)%frame_type() == TGM_FRAME_TYPE_NTENSOR, 'buffered[5] ntensor') + + ! An unhashed message carries no aggregate hash frame at all. + call buffered(1, .false., wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'unhashed walk: clean end') + call assert(n == 3, 'unhashed 1-object message: 3 frames') + call assert(frames(1)%frame_type() == TGM_FRAME_TYPE_HEADER_METADATA, 'unhashed[1] header metadata') + call assert(frames(2)%frame_type() == TGM_FRAME_TYPE_HEADER_INDEX, 'unhashed[2] header index') + call assert(frames(3)%frame_type() == TGM_FRAME_TYPE_NTENSOR, 'unhashed[3] ntensor') + end subroutine buffered_frame_sequence + + ! ---- Offsets / lengths: in bounds, ordered, non-overlapping ------------- + subroutine frame_geometry() + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_frame) :: frames(MAXF) + integer(c_int) :: err + integer(c_size_t) :: prev_end, first, last + integer :: n, k + + call buffered(2, .true., wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'geometry: clean end') + call assert(n > 0, 'geometry: some frames') + + ! %offset() is a 1-BASED Fortran index (the tensogram_scan convention), + ! so the first frame starts just past the 24-byte preamble. + prev_end = int(PREAMBLE, c_size_t) ! last byte of the preamble + do k = 1, n + first = frames(k)%offset() + last = first + frames(k)%length() - 1_c_size_t + call assert(first > prev_end, 'geometry: frames do not overlap') + call assert(last <= size(wire, kind=c_size_t), 'geometry: frame stays in bounds') + call assert(frames(k)%length() >= int(FRAME_HEADER, c_size_t), 'geometry: frame spans its header') + ! The whole-frame span always ends on the ENDF sentinel. + call assert(tag_at(wire, last - 3_c_size_t) == 'ENDF', 'geometry: length spans the frame footer') + prev_end = last + end do + call assert(prev_end <= size(wire, kind=c_size_t) - int(POSTAMBLE, c_size_t), & + 'geometry: the postamble is not a frame') + call assert(frames(1)%offset() == int(PREAMBLE + 1, c_size_t), 'geometry: first frame follows the preamble') + end subroutine frame_geometry + + ! ---- Payload span, per-frame hash flag, version ------------------------- + subroutine payload_and_hash_flags() + integer(c_int8_t), allocatable :: wire(:), payload(:) + type(tensogram_frame) :: frames(MAXF) + integer(c_int) :: err + integer(c_size_t) :: want, start + integer :: n, k, footer + + call buffered(2, .true., wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'payload: clean end') + do k = 1, n + if (frames(k)%frame_type() == TGM_FRAME_TYPE_NTENSOR) then + footer = FOOTER_NTEN + else + footer = FOOTER_ANY + end if + payload = frames(k)%payload() + want = frames(k)%length() - int(FRAME_HEADER + footer, c_size_t) + call assert(size(payload, kind=c_size_t) == want, 'payload: length - header - footer') + ! The copy holds exactly the bytes between header and footer. + start = frames(k)%offset() + int(FRAME_HEADER, c_size_t) + call assert(bytes_eq(payload, wire(start:start + want - 1_c_size_t)), 'payload: copied from the message') + call assert(frames(k)%has_hash(), 'hashed message: every frame has HASH_PRESENT') + call assert(iand(frames(k)%flags(), HASH_PRESENT) /= 0, 'hashed message: flag bit 1 agrees') + call assert(frames(k)%version() == 1, 'frame version 1') + end do + + call buffered(1, .false., wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'unhashed payload: clean end') + do k = 1, n + call assert(.not. frames(k)%has_hash(), 'unhashed message: no HASH_PRESENT') + call assert(iand(frames(k)%flags(), HASH_PRESENT) == 0, 'unhashed message: flag bit 1 clear') + end do + end subroutine payload_and_hash_flags + + ! ---- The payload copy survives the iterator (it is a copy, not a view) -- + subroutine payload_outlives_the_iterator() + integer(c_int8_t), allocatable :: wire(:), payload(:) + type(tensogram_frame) :: frames(MAXF) + integer(c_int) :: err + integer :: n + + call buffered(1, .true., wire) + call walk(wire, frames, n, err) ! walk() frees the iterator + call assert(n > 0, 'payload copy: some frames') + payload = frames(1)%payload() + call assert(size(payload) > 0, 'payload copy: readable after the iterator is gone') + call assert(bytes_eq(payload, wire(frames(1)%offset() + int(FRAME_HEADER, c_size_t): & + frames(1)%offset() + int(FRAME_HEADER, c_size_t) + & + size(payload, kind=c_size_t) - 1_c_size_t)), & + 'payload copy: still equals the message bytes') + ! A frame is a plain value: assigning it copies the payload too. + frames(2) = frames(1) + call assert(bytes_eq(frames(2)%payload(), payload), 'frame value type: copy-assignable') + end subroutine payload_outlives_the_iterator + + ! ---- A streaming message puts metadata / index / hash in the footer ----- + subroutine streaming_frame_sequence() + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_frame) :: frames(MAXF) + integer(c_int) :: err + integer :: n + + call streamed(wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'streaming walk: clean end') + call assert(n == 5, 'streamed message: 5 frames') + call assert(frames(1)%frame_type() == TGM_FRAME_TYPE_HEADER_METADATA, 'streamed[1] header metadata') + call assert(frames(2)%frame_type() == TGM_FRAME_TYPE_NTENSOR, 'streamed[2] ntensor') + call assert(frames(3)%frame_type() == TGM_FRAME_TYPE_FOOTER_METADATA, 'streamed[3] footer metadata') + call assert(frames(4)%frame_type() == TGM_FRAME_TYPE_FOOTER_HASH, 'streamed[4] footer hash') + call assert(frames(5)%frame_type() == TGM_FRAME_TYPE_FOOTER_INDEX, 'streamed[5] footer index') + end subroutine streaming_frame_sequence + + ! ---- End vs malformed: both stop the walk, only one sets `err` ---------- + subroutine end_versus_malformed() + integer(c_int8_t), allocatable :: wire(:), cut(:) + type(tensogram_frame) :: frames(MAXF), fr + type(tensogram_frame_iterator) :: it + integer(c_int) :: err, e2 + integer :: n, yielded + integer(c_size_t) :: chop + logical :: found + + call buffered(2, .true., wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'clean end: err stays OK') + + ! Advancing an exhausted cursor keeps reporting a clean end. + call tensogram_frames(wire, it, err) + call assert(err == TGM_ERROR_OK, 'clean end: iterator created') + do + call it%next(fr, found, e2) + if (.not. found) exit + end do + call assert(e2 == TGM_ERROR_OK, 'clean end: no error code') + call it%next(fr, found, e2) + call assert(.not. found .and. e2 == TGM_ERROR_OK, 'clean end: still ended, still no error') + call it%free() + + ! Cut INSIDE the last frame: the preamble still parses, the chain does not. + chop = frames(n)%offset() + 8_c_size_t + cut = wire(1:chop) + call tensogram_frames(cut, it, err) + call assert(err == TGM_ERROR_OK, 'truncated: preamble still parses') + yielded = 0 + do + call it%next(fr, found, e2) + if (.not. found) exit + yielded = yielded + 1 + end do + call assert(yielded > 0, 'truncated: the intact frames are still yielded') + call assert(e2 /= TGM_ERROR_OK, 'truncated: malformed chain sets err') + call assert(len(tensogram_last_error()) > 0, 'truncated: a reason is recorded') + call it%free() + + ! next() on a freed / never-created iterator is an argument error. + call it%next(fr, found, e2) + call assert(.not. found .and. e2 == TGM_ERROR_INVALID_ARG, 'next on a dead iterator -> INVALID_ARG') + + ! `err` is optional: without it the walk still runs, it just cannot tell + ! a clean end from a malformed chain (both simply stop the loop). + call tensogram_frames(wire, it, err) + call assert(err == TGM_ERROR_OK, 'next without err: iterator created') + yielded = 0 + do + call it%next(fr, found) + if (.not. found) exit + yielded = yielded + 1 + end do + call assert(yielded == n, 'next without err: same frames yielded') + call it%free() + end subroutine end_versus_malformed + + ! ---- Buffers that are not messages are rejected at creation ------------- + subroutine bad_buffers_rejected() + integer(c_int8_t), allocatable :: wire(:), junk(:), empty(:) + type(tensogram_frame_iterator) :: it + type(tensogram_message_header) :: hdr + integer(c_int) :: err + integer :: k + + allocate(junk(40)) + do k = 1, 40 + junk(k) = int(mod(k, 97), c_int8_t) + end do + call tensogram_frames(junk, it, err) + call assert(err /= TGM_ERROR_OK, 'junk buffer: no iterator') + call tensogram_message_header_read(junk, hdr, err) + call assert(err /= TGM_ERROR_OK, 'junk buffer: no header') + + allocate(empty(0)) + call tensogram_frames(empty, it, err) + call assert(err == TGM_ERROR_INVALID_ARG, 'empty buffer -> INVALID_ARG') + call tensogram_message_header_read(empty, hdr, err) + call assert(err == TGM_ERROR_INVALID_ARG, 'empty buffer header -> INVALID_ARG') + + call buffered(1, .true., wire) + call tensogram_frames(wire(1:8), it, err) + call assert(err /= TGM_ERROR_OK, 'truncated preamble: no iterator') + call tensogram_message_header_read(wire(1:8), hdr, err) + call assert(err /= TGM_ERROR_OK, 'truncated preamble: no header') + end subroutine bad_buffers_rejected + + ! ---- Header flags match the frames of a BUFFERED message exactly -------- + subroutine message_header_buffered() + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_frame) :: frames(MAXF) + type(tensogram_message_header) :: hdr + integer(c_int) :: err + integer :: n + + call buffered(2, .true., wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'header: walk ok') + call tensogram_message_header_read(wire, hdr, err) + call assert(err == TGM_ERROR_OK, 'header: read ok') + + call assert(hdr%version() == TGM_WIRE_VERSION, 'header: wire version') + call assert(hdr%total_length() == size(wire, kind=c_int64_t), 'header: total_length is the message size') + + call assert(hdr%has_header_metadata() .eqv. seen(frames, n, TGM_FRAME_TYPE_HEADER_METADATA), & + 'header: has_header_metadata matches the frames') + call assert(hdr%has_footer_metadata() .eqv. seen(frames, n, TGM_FRAME_TYPE_FOOTER_METADATA), & + 'header: has_footer_metadata matches the frames') + call assert(hdr%has_header_index() .eqv. seen(frames, n, TGM_FRAME_TYPE_HEADER_INDEX), & + 'header: has_header_index matches the frames') + call assert(hdr%has_footer_index() .eqv. seen(frames, n, TGM_FRAME_TYPE_FOOTER_INDEX), & + 'header: has_footer_index matches the frames') + call assert(hdr%has_header_hashes() .eqv. seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), & + 'header: has_header_hashes matches the frames') + call assert(hdr%has_footer_hashes() .eqv. seen(frames, n, TGM_FRAME_TYPE_FOOTER_HASH), & + 'header: has_footer_hashes matches the frames') + call assert(hdr%has_preceder_metadata() .eqv. seen(frames, n, TGM_FRAME_TYPE_PRECEDER_METADATA), & + 'header: has_preceder_metadata matches the frames') + call assert(hdr%has_hashes_present(), 'header: the default encode hashes every frame') + + ! Random access: metadata and index live in the header, not the footer. + call assert(hdr%has_header_metadata() .and. .not. hdr%has_footer_metadata(), 'header: random-access metadata') + call assert(hdr%has_header_index() .and. .not. hdr%has_footer_index(), 'header: random-access index') + + ! An unhashed message advertises neither aggregate hashes nor per-frame ones. + call buffered(1, .false., wire) + call tensogram_message_header_read(wire, hdr, err) + call assert(err == TGM_ERROR_OK, 'unhashed header: read ok') + call assert(.not. hdr%has_header_hashes(), 'unhashed header: no header hash frame') + call assert(.not. hdr%has_footer_hashes(), 'unhashed header: no footer hash frame') + call assert(.not. hdr%has_hashes_present(), 'unhashed header: no per-frame hashes') + end subroutine message_header_buffered + + ! ---- Streaming: only "frame present => flag set" is guaranteed ---------- + subroutine message_header_streaming() + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_frame) :: frames(MAXF) + type(tensogram_message_header) :: hdr + integer(c_int) :: err + integer :: n + + call streamed(wire) + call walk(wire, frames, n, err) + call assert(err == TGM_ERROR_OK, 'streaming header: walk ok') + call tensogram_message_header_read(wire, hdr, err) + call assert(err == TGM_ERROR_OK, 'streaming header: read ok') + + call assert(hdr%version() == TGM_WIRE_VERSION, 'streaming header: wire version') + call assert(implies(seen(frames, n, TGM_FRAME_TYPE_HEADER_METADATA), hdr%has_header_metadata()), & + 'streaming header: header metadata frame => flag') + call assert(implies(seen(frames, n, TGM_FRAME_TYPE_FOOTER_METADATA), hdr%has_footer_metadata()), & + 'streaming header: footer metadata frame => flag') + call assert(implies(seen(frames, n, TGM_FRAME_TYPE_HEADER_INDEX), hdr%has_header_index()), & + 'streaming header: header index frame => flag') + call assert(implies(seen(frames, n, TGM_FRAME_TYPE_FOOTER_INDEX), hdr%has_footer_index()), & + 'streaming header: footer index frame => flag') + call assert(implies(seen(frames, n, TGM_FRAME_TYPE_HEADER_HASH), hdr%has_header_hashes()), & + 'streaming header: header hash frame => flag') + call assert(implies(seen(frames, n, TGM_FRAME_TYPE_FOOTER_HASH), hdr%has_footer_hashes()), & + 'streaming header: footer hash frame => flag') + call assert(implies(seen(frames, n, TGM_FRAME_TYPE_PRECEDER_METADATA), hdr%has_preceder_metadata()), & + 'streaming header: preceder frame => flag') + ! Streaming mode: metadata + index land in the footer, and the + ! non-seeking sink never back-fills the preamble's total_length. + call assert(hdr%has_footer_metadata(), 'streaming header: footer metadata') + call assert(hdr%has_footer_index(), 'streaming header: footer index') + call assert(hdr%total_length() == 0_c_int64_t, 'streaming header: total_length never back-filled') + end subroutine message_header_streaming + + ! ---- helpers ------------------------------------------------------------ + + !> Walk `wire` to exhaustion, collecting the frames. `err` is the walk's + !> final status: OK for a clean end, non-OK for a malformed chain. + subroutine walk(wire, frames, n, err) + integer(c_int8_t), intent(in) :: wire(:) + type(tensogram_frame), intent(out) :: frames(:) + integer, intent(out) :: n + integer(c_int), intent(out) :: err + type(tensogram_frame_iterator) :: it + type(tensogram_frame) :: fr + logical :: found + n = 0 + call tensogram_frames(wire, it, err) + if (err /= TGM_ERROR_OK) return + do + call it%next(fr, found, err) + if (.not. found) exit + n = n + 1 + call assert(n <= size(frames), 'walk: frame buffer overflow') + frames(n) = fr + end do + call it%free() + end subroutine walk + + !> .true. when a frame of type `ftype` was yielded. + logical function seen(frames, n, ftype) + type(tensogram_frame), intent(in) :: frames(:) + integer, intent(in) :: n + integer(c_int), intent(in) :: ftype + integer :: k + seen = .false. + do k = 1, n + if (frames(k)%frame_type() == ftype) seen = .true. + end do + end function seen + + pure logical function implies(p, q) + logical, intent(in) :: p, q + implies = (.not. p) .or. q + end function implies + + !> The four ASCII bytes of `wire` starting at the 1-based index `at`. + function tag_at(wire, at) result(s) + integer(c_int8_t), intent(in) :: wire(:) + integer(c_size_t), intent(in) :: at + character(len=4) :: s + integer :: k + do k = 1, 4 + s(k:k) = achar(iand(int(wire(at + int(k - 1, c_size_t))), 255)) + end do + end function tag_at + + !> A buffered (random-access) message of `nobj` float32 objects, hashed or + !> not: metadata / index / hash frames all live in the header. + subroutine buffered(nobj, hashed, wire) + integer, intent(in) :: nobj + logical, intent(in) :: hashed + integer(c_int8_t), allocatable, intent(out) :: wire(:) + real(c_float) :: vals(4) + integer(c_int8_t), allocatable :: data(:) + integer(c_size_t), allocatable :: lens(:) + character(len=:), allocatable :: json + type(tensogram_buffer) :: buf + integer(c_int) :: err + integer :: k + vals = [1.0_c_float, 2.0_c_float, 3.0_c_float, 4.0_c_float] + allocate(lens(nobj), data(0)) + json = '{"descriptors":[' + do k = 1, nobj + if (k > 1) json = json // ',' + json = json // desc1d(4) + data = [data, transfer(vals, [0_c_int8_t], 4 * 4)] + lens(k) = int(4 * 4, c_size_t) + end do + json = json // ']}' + if (hashed) then + call tensogram_encode_pre_encoded(json, data, lens, buf, err) + else + call tensogram_encode_pre_encoded(json, data, lens, buf, err, hash='') + end if + call assert(err == TGM_ERROR_OK, 'fixture: buffered encode') + call buf%as_array(wire) + end subroutine buffered + + !> A streaming message (one object): metadata / index / hashes land in the + !> FOOTER and total_length is never back-filled. + subroutine streamed(wire) + integer(c_int8_t), allocatable, intent(out) :: wire(:) + character(len=*), parameter :: path = 'test_frames_stream.tgm' + real(c_float) :: vals(4) + type(tensogram_streaming_encoder) :: enc + type(tensogram_file) :: f + type(tensogram_buffer) :: buf + integer(c_int) :: err + integer :: unit, ios + vals = [1.0_c_float, 2.0_c_float, 3.0_c_float, 4.0_c_float] + call tensogram_streaming_encoder_create(path, enc, err) + call assert(err == TGM_ERROR_OK, 'fixture: stream create') + call tensogram_streaming_encoder_write(enc, vals, err) + call assert(err == TGM_ERROR_OK, 'fixture: stream write') + call tensogram_streaming_encoder_finish(enc, err) + call assert(err == TGM_ERROR_OK, 'fixture: stream finish') + call enc%free() + call tensogram_file_open(path, f, err) + call assert(err == TGM_ERROR_OK, 'fixture: stream reopen') + call tensogram_file_read_message(f, 1, buf, err) + call assert(err == TGM_ERROR_OK, 'fixture: stream read') + call buf%as_array(wire) + call f%close() + open(newunit=unit, file=path, status='old', iostat=ios) + if (ios == 0) close(unit, status='delete') + end subroutine streamed + + !> A bare 1-D float32 descriptor (no encoding pipeline) for `n` elements. + function desc1d(n) result(s) + integer, intent(in) :: n + character(len=:), allocatable :: s + character(len=32) :: ns + write (ns, '(i0)') n + s = '{"type":"ndarray","ndim":1,"shape":[' // trim(ns) // & + '],"strides":[4],"dtype":"float32","byte_order":"' // host_bo() // & + '","encoding":"none","filter":"none","compression":"none"}' + end function desc1d + + !> Host byte order as the wire descriptor spells it ("little" / "big"). + function host_bo() result(bo) + character(len=:), allocatable :: bo + integer(c_int8_t) :: probe(4) + probe = transfer(1_c_int32_t, 0_c_int8_t, 4) + if (probe(1) == 1_c_int8_t) then + bo = 'little' + else + bo = 'big' + end if + end function host_bo + + logical function bytes_eq(x, y) + integer(c_int8_t), intent(in) :: x(:), y(:) + bytes_eq = size(x) == size(y) + if (bytes_eq) bytes_eq = all(x == y) + end function bytes_eq + + subroutine assert(cond, what) + logical, intent(in) :: cond + character(len=*), intent(in) :: what + if (.not. cond) then + print '(a,a)', 'test_frames: FAIL: ', what + error stop 1 + end if + npass = npass + 1 + end subroutine assert + +end program test_frames diff --git a/fortran/test/test_guards.f90 b/fortran/test/test_guards.f90 index 228e7acb..b5ba5502 100644 --- a/fortran/test/test_guards.f90 +++ b/fortran/test/test_guards.f90 @@ -26,6 +26,7 @@ program test_guards case ('file'); call g_file() case ('metadata'); call g_metadata() case ('stream'); call g_stream() + case ('frames'); call g_frames() case ('check_ctx'); call tensogram_check(TGM_ERROR_OBJECT, 'guard-context') case ('check_noctx'); call tensogram_check(TGM_ERROR_OBJECT) case default; print '(a)', 'unknown guard mode'; error stop 2 @@ -92,4 +93,18 @@ subroutine g_stream() if (tensogram_streaming_encoder_count(b) < 0) print '(a)', 'x' end subroutine g_stream + subroutine g_frames() + type(tensogram_buffer) :: buf + type(tensogram_frame_iterator) :: a, b + type(tensogram_frame) :: fr + integer(c_int8_t), allocatable :: w(:) + integer(c_int) :: err + logical :: found + call live_buffer(buf); call buf%as_array(w) + call tensogram_frames(w, a, err) + b = a ! non-copyable guard -> error stop + call b%next(fr, found) + if (found) print '(a)', 'x' + end subroutine g_frames + end program test_guards diff --git a/fortran/test/test_remote.f90 b/fortran/test/test_remote.f90 new file mode 100644 index 00000000..ced0cfbd --- /dev/null +++ b/fortran/test/test_remote.f90 @@ -0,0 +1,184 @@ +! (C) Copyright 2026- ECMWF and individual contributors. +! +! This software is licensed under the terms of the Apache Licence Version 2.0 +! which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +! In applying this licence, ECMWF does not waive the privileges and immunities +! granted to it by virtue of its status as an intergovernmental organisation nor +! does it submit to any jurisdiction. + +!> Deferred Wave-B, component B — the SYNCHRONOUS remote surface: +!> * tensogram_is_remote_url (tgm_is_remote_url) +!> * tensogram_file_open_remote (tgm_file_open_remote) +!> +!> The remote backend is an opt-in Cargo feature of the C library, but its +!> symbols always link: a build without `--features=remote` answers "not remote +!> for me" to every URL and returns TGM_ERROR_REMOTE (with an explanatory +!> message) from open_remote. Both flavours are covered here — the build's own +!> answer for one object-store URL selects which expectations apply, so the +!> test is honest on a feature-off build and still exercises the real backend +!> when one is present. +program test_remote + use, intrinsic :: iso_c_binding + use tensogram + implicit none + + integer :: npass + logical :: remote_capable + + npass = 0 + ! One probe decides this build's flavour; every other scheme must agree. + remote_capable = tensogram_is_remote_url('s3://bucket/key.tgm') + + call local_sources_are_never_remote() + call object_store_schemes_match_the_build() + call open_remote_validates_arguments_first() + call open_remote_behaviour() + + print '(a,i0,a,l1,a)', 'test_remote: PASS (', npass, ' checks, remote feature = ', remote_capable, ')' + +contains + + ! ---- Local sources belong to the local backend, in every build ---------- + subroutine local_sources_are_never_remote() + call assert(.not. tensogram_is_remote_url('/tmp/data.tgm'), 'absolute path is not remote') + call assert(.not. tensogram_is_remote_url('data.tgm'), 'bare filename is not remote') + call assert(.not. tensogram_is_remote_url('./relative/data.tgm'), 'relative path is not remote') + call assert(.not. tensogram_is_remote_url('file:///tmp/data.tgm'), 'file:// is not remote') + call assert(.not. tensogram_is_remote_url(''), 'empty source is not remote') + call assert(.not. tensogram_is_remote_url('ftp://host/data.tgm'), 'ftp:// is not a supported scheme') + end subroutine local_sources_are_never_remote + + ! ---- Every object-store scheme answers with this build's capability ---- + subroutine object_store_schemes_match_the_build() + call assert(tensogram_is_remote_url('s3://bucket/key.tgm') .eqv. remote_capable, 's3://') + call assert(tensogram_is_remote_url('s3a://bucket/key.tgm') .eqv. remote_capable, 's3a://') + call assert(tensogram_is_remote_url('gs://bucket/key.tgm') .eqv. remote_capable, 'gs://') + call assert(tensogram_is_remote_url('az://container/key.tgm') .eqv. remote_capable, 'az://') + call assert(tensogram_is_remote_url('azure://container/key.tgm').eqv. remote_capable, 'azure://') + call assert(tensogram_is_remote_url('http://host/key.tgm') .eqv. remote_capable, 'http://') + call assert(tensogram_is_remote_url('https://host/key.tgm') .eqv. remote_capable, 'https://') + ! Schemes are compared case-insensitively. + call assert(tensogram_is_remote_url('S3://bucket/key.tgm') .eqv. remote_capable, 'S3:// (upper case)') + call assert(tensogram_is_remote_url('HTTPS://host/key.tgm') .eqv. remote_capable, 'HTTPS:// (upper case)') + ! Trailing blanks of a fixed-length actual argument are not part of the URL. + call assert(tensogram_is_remote_url(padded()) .eqv. remote_capable, 'blank-padded source is trimmed') + end subroutine object_store_schemes_match_the_build + + ! ---- Argument validation is identical in every build -------------------- + ! Parallel arrays are only well-formed when they are the same length; the + ! C ABI takes a single n_options, so the mismatch is caught here rather + ! than silently reading past the shorter array. + subroutine open_remote_validates_arguments_first() + type(tensogram_file) :: f + integer(c_int) :: err + character(len=16) :: keys2(2), values1(1), keys1(1), values2(2), none(0) + keys2 = ['aws_region ', 'aws_endpoint '] + values1 = ['eu-west-1 '] + keys1 = ['aws_region '] + values2 = ['eu-west-1 ', 'http://localhost'] + + ! Rejected before the C call, so no build ever touches the network here. + call tensogram_file_open_remote('s3://bucket/key.tgm', keys2, values1, f, err) + call assert(err == TGM_ERROR_INVALID_ARG, 'more keys than values -> INVALID_ARG') + call tensogram_file_open_remote('s3://bucket/key.tgm', keys1, values2, f, err) + call assert(err == TGM_ERROR_INVALID_ARG, 'more values than keys -> INVALID_ARG') + ! Zero-size arrays mean "no storage options" — never an argument error. + ! A `file://` URL keeps this hermetic: it reaches the backend (or the + ! feature-off stub) without a network round-trip in either build. + call tensogram_file_open_remote('file:///nonexistent/missing.tgm', none, none, f, err) + call assert(err /= TGM_ERROR_INVALID_ARG, 'empty option arrays are not an argument error') + end subroutine open_remote_validates_arguments_first + + ! ---- open_remote: feature-off explains itself, feature-on reads --------- + subroutine open_remote_behaviour() + type(tensogram_file) :: f + integer(c_int) :: err + character(len=16) :: keys(1), values(1) + character(len=:), allocatable :: detail + keys = ['aws_region '] + values = ['eu-west-1 '] + + if (.not. remote_capable) then + ! Documented feature-off contract: TGM_ERROR_REMOTE + how to fix it. + call tensogram_file_open_remote('s3://bucket/key.tgm', f, err) + call assert(err == TGM_ERROR_REMOTE, 'feature-off open_remote -> TGM_ERROR_REMOTE') + detail = tensogram_last_error() + call assert(index(detail, 'remote') > 0, 'feature-off open_remote explains the missing feature') + call assert(index(detail, 'features=remote') > 0, 'feature-off open_remote says how to enable it') + + ! Storage options and scan options are marshalled before the feature + ! check, so a well-formed call still reaches the same honest answer. + call tensogram_file_open_remote('s3://bucket/key.tgm', keys, values, f, err) + call assert(err == TGM_ERROR_REMOTE, 'feature-off open_remote with options -> TGM_ERROR_REMOTE') + call tensogram_file_open_remote('s3://bucket/key.tgm', f, err, bidirectional=.false.) + call assert(err == TGM_ERROR_REMOTE, 'feature-off open_remote forward-only -> TGM_ERROR_REMOTE') + else + call open_remote_round_trip() + end if + end subroutine open_remote_behaviour + + !> A remote-capable build: `file://` travels the same object-store code + !> path as `s3://` & friends, so the round-trip needs no network. + subroutine open_remote_round_trip() + character(len=*), parameter :: path = 'test_remote_fixture.tgm' + character(len=4096) :: cwd + character(len=:), allocatable :: url + type(tensogram_file) :: f, remote + type(tensogram_message) :: msg + real(c_float), allocatable :: got(:) + integer(c_int) :: err + integer :: n, unit, ios + call get_environment_variable('PWD', cwd, status=ios) + if (ios /= 0 .or. len_trim(cwd) == 0) return ! cannot build an absolute URL + url = 'file://' // trim(cwd) // '/' // path + + call tensogram_file_create(path, f, err); call assert(err == TGM_ERROR_OK, 'fixture: create') + call tensogram_file_append(f, [1.0_c_float, 2.0_c_float, 3.0_c_float], err) + call assert(err == TGM_ERROR_OK, 'fixture: append A') + call tensogram_file_append(f, [5.0_c_float, 6.0_c_float], err) + call assert(err == TGM_ERROR_OK, 'fixture: append B') + call f%close() + + ! The returned handle is an ordinary tensogram_file: the whole file API works. + call tensogram_file_open_remote(url, remote, err) + call assert(err == TGM_ERROR_OK, 'open_remote file:// -> OK') + call tensogram_file_message_count(remote, n, err) + call assert(err == TGM_ERROR_OK .and. n == 2, 'open_remote: two messages') + call tensogram_file_decode_message(remote, 2, msg, err) + call assert(err == TGM_ERROR_OK, 'open_remote: decode message 2') + call tensogram_to_array(msg, 1, got, err) + call assert(err == TGM_ERROR_OK .and. size(got) == 2, 'open_remote: object round-trips') + call remote%close() + + ! Forward-only walk is accepted too. + call tensogram_file_open_remote(url, remote, err, bidirectional=.false.) + call assert(err == TGM_ERROR_OK, 'open_remote forward-only -> OK') + call remote%close() + + ! A missing object is a remote error, not a local I/O error. + call tensogram_file_open_remote('file:///nonexistent/missing.tgm', remote, err) + call assert(err /= TGM_ERROR_OK, 'open_remote missing object -> error') + + open(newunit=unit, file=path, status='old', iostat=ios) + if (ios == 0) close(unit, status='delete') + end subroutine open_remote_round_trip + + ! ---- helpers ------------------------------------------------------------ + + !> A blank-padded fixed-length source, as an application would pass one. + function padded() result(s) + character(len=32) :: s + s = 's3://bucket/key.tgm' + end function padded + + subroutine assert(cond, what) + logical, intent(in) :: cond + character(len=*), intent(in) :: what + if (.not. cond) then + print '(a,a)', 'test_remote: FAIL: ', what + error stop 1 + end if + npass = npass + 1 + end subroutine assert + +end program test_remote diff --git a/fortran/test/test_typed_enums.f90 b/fortran/test/test_typed_enums.f90 new file mode 100644 index 00000000..6a44e643 --- /dev/null +++ b/fortran/test/test_typed_enums.f90 @@ -0,0 +1,245 @@ +! (C) Copyright 2026- ECMWF and individual contributors. +! +! This software is licensed under the terms of the Apache Licence Version 2.0 +! which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +! In applying this licence, ECMWF does not waive the privileges and immunities +! granted to it by virtue of its status as an intergovernmental organisation nor +! does it submit to any jurisdiction. + +!> Deferred Wave-B, component C — the typed enum surface: +!> * TGM_DTYPE_* / TGM_BYTE_ORDER_* (tgm_dtype/tgm_byte_order) +!> * TGM_AGGREGATE_HASH_POLICY_* / TGM_COMPRESSION_BACKEND_* +!> * tensogram_object_dtype_enum (tgm_object_dtype_enum) +!> * tensogram_object_byte_order_enum (tgm_object_byte_order_enum) +!> +!> The parameter values themselves are guarded against the C header by +!> fortran/test/check_enum_mirror.sh; what a running test adds is that the +!> CODES agree with the STRING getters they are the typed companion of, for +!> every dtype this binding can put on the wire. +program test_typed_enums + use, intrinsic :: iso_c_binding + use tensogram + implicit none + + integer :: npass + + npass = 0 + + call dtype_constants() + call byte_order_constants() + call policy_constants() + call dtype_enum_agrees_with_the_string_getter() + call byte_order_enum_agrees_with_the_string_getter() + call out_of_range_is_reported() + + print '(a,i0,a)', 'test_typed_enums: PASS (', npass, ' checks)' + +contains + + ! ---- The mirrored C ABI numbers (frozen; the wire stores dtype as text) - + subroutine dtype_constants() + call assert(TGM_DTYPE_FLOAT16 == 0, 'TGM_DTYPE_FLOAT16 = 0') + call assert(TGM_DTYPE_BFLOAT16 == 1, 'TGM_DTYPE_BFLOAT16 = 1') + call assert(TGM_DTYPE_FLOAT32 == 2, 'TGM_DTYPE_FLOAT32 = 2') + call assert(TGM_DTYPE_FLOAT64 == 3, 'TGM_DTYPE_FLOAT64 = 3') + call assert(TGM_DTYPE_COMPLEX64 == 4, 'TGM_DTYPE_COMPLEX64 = 4') + call assert(TGM_DTYPE_COMPLEX128 == 5, 'TGM_DTYPE_COMPLEX128 = 5') + call assert(TGM_DTYPE_INT8 == 6, 'TGM_DTYPE_INT8 = 6') + call assert(TGM_DTYPE_INT16 == 7, 'TGM_DTYPE_INT16 = 7') + call assert(TGM_DTYPE_INT32 == 8, 'TGM_DTYPE_INT32 = 8') + call assert(TGM_DTYPE_INT64 == 9, 'TGM_DTYPE_INT64 = 9') + call assert(TGM_DTYPE_UINT8 == 10, 'TGM_DTYPE_UINT8 = 10') + call assert(TGM_DTYPE_UINT16 == 11, 'TGM_DTYPE_UINT16 = 11') + call assert(TGM_DTYPE_UINT32 == 12, 'TGM_DTYPE_UINT32 = 12') + call assert(TGM_DTYPE_UINT64 == 13, 'TGM_DTYPE_UINT64 = 13') + call assert(TGM_DTYPE_BITMASK == 14, 'TGM_DTYPE_BITMASK = 14') + end subroutine dtype_constants + + subroutine byte_order_constants() + call assert(TGM_BYTE_ORDER_LITTLE == 0, 'TGM_BYTE_ORDER_LITTLE = 0') + call assert(TGM_BYTE_ORDER_BIG == 1, 'TGM_BYTE_ORDER_BIG = 1') + end subroutine byte_order_constants + + subroutine policy_constants() + call assert(TGM_AGGREGATE_HASH_POLICY_AUTO == 0, 'AGGREGATE_HASH_POLICY_AUTO = 0') + call assert(TGM_AGGREGATE_HASH_POLICY_NONE == 1, 'AGGREGATE_HASH_POLICY_NONE = 1') + call assert(TGM_AGGREGATE_HASH_POLICY_HEADER == 2, 'AGGREGATE_HASH_POLICY_HEADER = 2') + call assert(TGM_AGGREGATE_HASH_POLICY_FOOTER == 3, 'AGGREGATE_HASH_POLICY_FOOTER = 3') + call assert(TGM_AGGREGATE_HASH_POLICY_BOTH == 4, 'AGGREGATE_HASH_POLICY_BOTH = 4') + call assert(TGM_COMPRESSION_BACKEND_AUTO == 0, 'COMPRESSION_BACKEND_AUTO = 0') + call assert(TGM_COMPRESSION_BACKEND_FFI == 1, 'COMPRESSION_BACKEND_FFI = 1') + call assert(TGM_COMPRESSION_BACKEND_PURE == 2, 'COMPRESSION_BACKEND_PURE = 2') + end subroutine policy_constants + + ! ---- dtype_enum is the typed companion of the dtype string -------------- + subroutine dtype_enum_agrees_with_the_string_getter() + character(len=8), parameter :: names(10) = & + ['float32 ', 'float64 ', 'int8 ', 'int16 ', 'int32 ', & + 'int64 ', 'uint8 ', 'uint16 ', 'uint32 ', 'uint64 '] + integer, parameter :: widths(10) = [4, 8, 1, 2, 4, 8, 1, 2, 4, 8] + integer(c_int8_t), allocatable :: data(:), wire(:) + integer(c_size_t), allocatable :: lens(:) + character(len=:), allocatable :: json + type(tensogram_buffer) :: buf + type(tensogram_message) :: msg + integer(c_int) :: err, code + integer :: k + + json = '{"descriptors":[' + allocate(lens(size(names)), data(0)) + do k = 1, size(names) + if (k > 1) json = json // ',' + json = json // desc1(trim(names(k)), widths(k)) + data = [data, spread(int(k, c_int8_t), 1, widths(k))] + lens(k) = int(widths(k), c_size_t) + end do + json = json // ']}' + + call tensogram_encode_pre_encoded(json, data, lens, buf, err) + call assert(err == TGM_ERROR_OK, 'dtype_enum: encode the dtype zoo') + call buf%as_array(wire) + call tensogram_decode(wire, msg, err) + call assert(err == TGM_ERROR_OK, 'dtype_enum: decode the dtype zoo') + call assert(tensogram_num_objects(msg) == size(names), 'dtype_enum: every object decoded') + + do k = 1, size(names) + code = tensogram_object_dtype_enum(msg, k, err) + call assert(err == TGM_ERROR_OK, 'dtype_enum: in-range index is not an error') + call assert(code == expected_dtype(tensogram_object_dtype(msg, k)), & + 'dtype_enum agrees with the dtype string: ' // trim(names(k))) + end do + end subroutine dtype_enum_agrees_with_the_string_getter + + ! ---- byte_order_enum likewise ------------------------------------------- + subroutine byte_order_enum_agrees_with_the_string_getter() + real(c_float) :: a(3) + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_buffer) :: buf + type(tensogram_message) :: msg + integer(c_int) :: err, code + a = [1.0_c_float, 2.0_c_float, 3.0_c_float] + call tensogram_encode(a, buf, err) + call assert(err == TGM_ERROR_OK, 'byte_order_enum: encode') + call buf%as_array(wire) + ! native_byte_order = .false. keeps the descriptor's own byte order. + call tensogram_decode(wire, msg, err, native_byte_order=.false.) + call assert(err == TGM_ERROR_OK, 'byte_order_enum: decode') + code = tensogram_object_byte_order_enum(msg, 1, err) + call assert(err == TGM_ERROR_OK, 'byte_order_enum: in-range index is not an error') + call assert(code == expected_byte_order(tensogram_object_byte_order(msg, 1)), & + 'byte_order_enum agrees with the byte-order string') + call assert(code == expected_byte_order(host_bo()), 'byte_order_enum reports the host order') + end subroutine byte_order_enum_agrees_with_the_string_getter + + ! ---- Out of range: the zero variant is ambiguous, so `err` says so ----- + subroutine out_of_range_is_reported() + real(c_float) :: a(2) + integer(c_int8_t), allocatable :: wire(:) + type(tensogram_buffer) :: buf + type(tensogram_message) :: msg, null_msg + integer(c_int) :: err, code + a = [1.0_c_float, 2.0_c_float] + call tensogram_encode(a, buf, err) + call assert(err == TGM_ERROR_OK, 'out of range: encode') + call buf%as_array(wire) + call tensogram_decode(wire, msg, err) + call assert(err == TGM_ERROR_OK, 'out of range: decode') + + code = tensogram_object_dtype_enum(msg, 2, err) + call assert(err == TGM_ERROR_INVALID_ARG, 'dtype_enum past the end -> INVALID_ARG') + call assert(code == TGM_DTYPE_FLOAT16, 'dtype_enum past the end -> zero variant') + call assert(len(tensogram_object_dtype(msg, 2)) == 0, 'the string getter agrees (empty)') + + code = tensogram_object_dtype_enum(msg, 0, err) + call assert(err == TGM_ERROR_INVALID_ARG, 'dtype_enum index 0 -> INVALID_ARG') + code = tensogram_object_byte_order_enum(msg, 2, err) + call assert(err == TGM_ERROR_INVALID_ARG, 'byte_order_enum past the end -> INVALID_ARG') + call assert(code == TGM_BYTE_ORDER_LITTLE, 'byte_order_enum past the end -> zero variant') + + ! A null (unassigned) handle has no objects at all. + code = tensogram_object_dtype_enum(null_msg, 1, err) + call assert(err == TGM_ERROR_INVALID_ARG, 'dtype_enum on a null handle -> INVALID_ARG') + code = tensogram_object_byte_order_enum(null_msg, 1, err) + call assert(err == TGM_ERROR_INVALID_ARG, 'byte_order_enum on a null handle -> INVALID_ARG') + + ! `err` is optional: the accessors are usable in an expression. + call assert(tensogram_object_dtype_enum(msg, 1) == TGM_DTYPE_FLOAT32, 'dtype_enum without err') + call assert(tensogram_object_byte_order_enum(msg, 1) == expected_byte_order(host_bo()), & + 'byte_order_enum without err') + end subroutine out_of_range_is_reported + + ! ---- helpers ------------------------------------------------------------ + + !> The TGM_DTYPE_* code a dtype string must map to (the test's own table, + !> so an accessor that silently returned the wrong variant is caught). + integer(c_int) function expected_dtype(name) + character(len=*), intent(in) :: name + select case (name) + case ('float16'); expected_dtype = TGM_DTYPE_FLOAT16 + case ('bfloat16'); expected_dtype = TGM_DTYPE_BFLOAT16 + case ('float32'); expected_dtype = TGM_DTYPE_FLOAT32 + case ('float64'); expected_dtype = TGM_DTYPE_FLOAT64 + case ('complex64'); expected_dtype = TGM_DTYPE_COMPLEX64 + case ('complex128'); expected_dtype = TGM_DTYPE_COMPLEX128 + case ('int8'); expected_dtype = TGM_DTYPE_INT8 + case ('int16'); expected_dtype = TGM_DTYPE_INT16 + case ('int32'); expected_dtype = TGM_DTYPE_INT32 + case ('int64'); expected_dtype = TGM_DTYPE_INT64 + case ('uint8'); expected_dtype = TGM_DTYPE_UINT8 + case ('uint16'); expected_dtype = TGM_DTYPE_UINT16 + case ('uint32'); expected_dtype = TGM_DTYPE_UINT32 + case ('uint64'); expected_dtype = TGM_DTYPE_UINT64 + case ('bitmask'); expected_dtype = TGM_DTYPE_BITMASK + case default + print '(a,a)', 'test_typed_enums: FAIL: unknown dtype string ', name + error stop 1 + end select + end function expected_dtype + + integer(c_int) function expected_byte_order(name) + character(len=*), intent(in) :: name + select case (name) + case ('little'); expected_byte_order = TGM_BYTE_ORDER_LITTLE + case ('big'); expected_byte_order = TGM_BYTE_ORDER_BIG + case default + print '(a,a)', 'test_typed_enums: FAIL: unknown byte order string ', name + error stop 1 + end select + end function expected_byte_order + + !> A single-element 1-D descriptor of `dtype` (`width` bytes per element, + !> no encoding pipeline, so the pre-encoded bytes ARE the element bytes). + function desc1(dtype, width) result(s) + character(len=*), intent(in) :: dtype + integer, intent(in) :: width + character(len=:), allocatable :: s + character(len=32) :: ws + write (ws, '(i0)') width + s = '{"type":"ndarray","ndim":1,"shape":[1],"strides":[' // trim(ws) // & + '],"dtype":"' // dtype // '","byte_order":"' // host_bo() // & + '","encoding":"none","filter":"none","compression":"none"}' + end function desc1 + + !> Host byte order as the wire descriptor spells it ("little" / "big"). + function host_bo() result(bo) + character(len=:), allocatable :: bo + integer(c_int8_t) :: probe(4) + probe = transfer(1_c_int32_t, 0_c_int8_t, 4) + if (probe(1) == 1_c_int8_t) then + bo = 'little' + else + bo = 'big' + end if + end function host_bo + + subroutine assert(cond, what) + logical, intent(in) :: cond + character(len=*), intent(in) :: what + if (.not. cond) then + print '(a,a)', 'test_typed_enums: FAIL: ', what + error stop 1 + end if + npass = npass + 1 + end subroutine assert + +end program test_typed_enums diff --git a/python/bindings/README.md b/python/bindings/README.md index b3b9e4e5..0142feb4 100644 --- a/python/bindings/README.md +++ b/python/bindings/README.md @@ -39,6 +39,8 @@ arr = result.objects[0][1] # numpy array - Sync and async file APIs (`TensogramFile` / `AsyncTensogramFile`) - GIL-free parallel encode / decode on free-threaded Python - Partial-range decode (`decode_range`) +- Structural introspection without decoding: the frame walker + (`tensogram.frames`) and the typed preamble (`tensogram.message_header`) - Full codec support: szip, zstd, lz4, blosc2, zfp, sz3 - Validation (`tensogram.validate`, `tensogram.validate_file`) - GRIB / NetCDF conversion (when the wheel is built with the matching diff --git a/python/bindings/python/tensogram/__init__.py b/python/bindings/python/tensogram/__init__.py index de8893ee..b597a21c 100644 --- a/python/bindings/python/tensogram/__init__.py +++ b/python/bindings/python/tensogram/__init__.py @@ -19,9 +19,14 @@ # the reader-side scan family (``scan_file`` / ``scan_with_options`` / # ``ScanOptions``), multi-message ``validate_buffer``, the lazy per-object # iterators (``objects`` / ``objects_metadata``), ``decode_range_from_payload``, -# and ``MessageLayout``. ``AsyncStreamingEncoder`` is feature-gated on the -# ``async`` build (default-on, like ``AsyncTensogramFile``) and is left to the -# star import so a no-async build still imports cleanly. +# ``MessageLayout``, and the structural walkers (``frames`` / ``Frame`` / +# ``FrameIter`` / ``message_header`` / ``MessageHeader``). +# ``AsyncStreamingEncoder`` is feature-gated on the ``async`` build +# (default-on, like ``AsyncTensogramFile``) and is left to the star import so a +# no-async build still imports cleanly. +from .tensogram import Frame as Frame +from .tensogram import FrameIter as FrameIter +from .tensogram import MessageHeader as MessageHeader from .tensogram import MessageLayout as MessageLayout from .tensogram import Metadata as Metadata from .tensogram import ObjectIter as ObjectIter @@ -30,6 +35,8 @@ from .tensogram import __version__ as __version__ from .tensogram import compute_common as compute_common from .tensogram import decode_range_from_payload as decode_range_from_payload +from .tensogram import frames as frames +from .tensogram import message_header as message_header from .tensogram import object_inline_hashes as object_inline_hashes from .tensogram import objects as objects from .tensogram import objects_metadata as objects_metadata diff --git a/python/bindings/src/lib.rs b/python/bindings/src/lib.rs index 7e2f44a7..b1616f9a 100644 --- a/python/bindings/src/lib.rs +++ b/python/bindings/src/lib.rs @@ -37,13 +37,14 @@ use tensogram_lib::validate::{ validate_file as core_validate_file, validate_message, }; use tensogram_lib::{ - ByteOrder, DataObjectDescriptor, DecodeOptions, Dtype, EncodeOptions, GlobalMetadata, - MessageLayout, ObjectIter as CoreObjectIter, RESERVED_KEY, RemoteScanOptions, + ByteOrder, DataObjectDescriptor, DecodeOptions, Dtype, EncodeOptions, FrameType, + GlobalMetadata, MessageLayout, ObjectIter as CoreObjectIter, RESERVED_KEY, RemoteScanOptions, ScanOptions as CoreScanOptions, StreamingEncoder, TensogramError, TensogramFile, compute_common, data_object_inline_hashes, decode, decode_descriptors, decode_metadata, decode_object, decode_range, decode_range_from_payload, encode, encode_pre_encoded, - objects as core_objects, objects_metadata as core_objects_metadata, parse_hash_name, scan, - scan_file, scan_file_with_options, scan_with_options, + frames as core_frames, message_header as core_message_header, objects as core_objects, + objects_metadata as core_objects_metadata, parse_hash_name, scan, scan_file, + scan_file_with_options, scan_with_options, }; type PyObject = Py; @@ -803,6 +804,10 @@ impl PyTensogramFile { /// optionally ``byte_order``, ``encoding``, ``filter``, ``compression``. /// hash: ``"xxh3"`` (default) or ``None`` to skip hashing. /// threads: thread budget (0 = sequential / env fallback). + /// aggregate_hash: aggregate hash frame placement — ``"auto"`` + /// (default), ``"none"``, ``"header"``, ``"footer"``, ``"both"``. + /// compression_backend: szip / zstd implementation — ``"auto"`` + /// (default), ``"ffi"``, ``"pure"``. #[pyo3( signature = ( global_meta_dict, @@ -816,6 +821,7 @@ impl PyTensogramFile { neg_inf_mask_method=None, small_mask_threshold_bytes=None, aggregate_hash=None, + compression_backend=None, ) )] #[allow(clippy::too_many_arguments)] @@ -833,6 +839,7 @@ impl PyTensogramFile { neg_inf_mask_method: Option<&str>, small_mask_threshold_bytes: Option, aggregate_hash: Option<&str>, + compression_backend: Option<&str>, ) -> PyResult<()> { let global_meta = dict_to_global_metadata(global_meta_dict)?; let pairs = extract_descriptor_data_pairs(py, descriptors_and_data)?; @@ -849,6 +856,7 @@ impl PyTensogramFile { neg_inf_mask_method, small_mask_threshold_bytes, aggregate_hash, + compression_backend, )?; py.detach(|| self.file.append(&global_meta, &refs, &options)) .map_err(to_py_err) @@ -1260,9 +1268,20 @@ impl PyFileIter { /// global_meta_dict: ``{"base": [...], ...}`` with any extra keys. /// descriptors_and_data: list of ``(descriptor_dict, numpy_array)`` pairs. /// hash: ``"xxh3"`` (default) or ``None`` to skip integrity hashing. +/// aggregate_hash: where to place the aggregate hash frame — +/// ``"auto"`` (default; buffered → header), ``"none"``, +/// ``"header"``, ``"footer"`` or ``"both"``. Verifiable with +/// :func:`frames` (``HeaderHash`` / ``FooterHash``). +/// compression_backend: which szip / zstd implementation to use when +/// both are compiled in — ``"auto"`` (default; consults +/// ``TENSOGRAM_COMPRESSION_BACKEND``), ``"ffi"`` or ``"pure"``. /// /// Returns: /// ``bytes`` — the complete wire-format message. +/// +/// Raises: +/// ValueError: On an unknown ``aggregate_hash`` or +/// ``compression_backend`` value. #[pyfunction] #[pyo3( name = "encode", @@ -1278,6 +1297,7 @@ impl PyFileIter { neg_inf_mask_method=None, small_mask_threshold_bytes=None, aggregate_hash=None, + compression_backend=None, ) )] #[allow(clippy::too_many_arguments)] @@ -1294,6 +1314,7 @@ fn py_encode<'py>( neg_inf_mask_method: Option<&str>, small_mask_threshold_bytes: Option, aggregate_hash: Option<&str>, + compression_backend: Option<&str>, ) -> PyResult> { let global_meta = dict_to_global_metadata(global_meta_dict)?; let pairs = extract_descriptor_data_pairs(py, descriptors_and_data)?; @@ -1310,6 +1331,7 @@ fn py_encode<'py>( neg_inf_mask_method, small_mask_threshold_bytes, aggregate_hash, + compression_backend, )?; let msg = py.detach(|| encode(&global_meta, &refs, &options).map_err(to_py_err))?; Ok(PyBytes::new(py, &msg)) @@ -2161,6 +2183,389 @@ impl PyObjectMetadataIter { } } +// --------------------------------------------------------------------------- +// Frame walker — Frame / FrameIter / MessageHeader +// --------------------------------------------------------------------------- + +/// Canonical wire-format name of a frame type (`plans/WIRE_FORMAT.md` §4). +/// +/// The name is the Python-facing identity of a frame type: it is what +/// :attr:`Frame.frame_type` returns, paired with the numeric wire code in +/// :attr:`Frame.frame_type_code`. Type 4 is reserved (the obsolete v2 +/// tensor frame) and never reaches this function — the core walker +/// rejects it while parsing the frame header. +fn frame_type_name(frame_type: FrameType) -> &'static str { + match frame_type { + FrameType::HeaderMetadata => "HeaderMetadata", + FrameType::HeaderIndex => "HeaderIndex", + FrameType::HeaderHash => "HeaderHash", + FrameType::FooterHash => "FooterHash", + FrameType::FooterIndex => "FooterIndex", + FrameType::FooterMetadata => "FooterMetadata", + FrameType::PrecederMetadata => "PrecederMetadata", + FrameType::NTensorFrame => "NTensorFrame", + } +} + +/// One frame of a message, as reported by :func:`frames`. +/// +/// Direct binding of the Rust core ``tensogram::FrameInfo``. All byte +/// positions are relative to the start of the message that was handed to +/// :func:`frames`. +/// +/// Attributes: +/// frame_type (str): Wire-format name — one of ``"HeaderMetadata"``, +/// ``"HeaderIndex"``, ``"HeaderHash"``, ``"FooterHash"``, +/// ``"FooterIndex"``, ``"FooterMetadata"``, ``"PrecederMetadata"``, +/// ``"NTensorFrame"``. +/// frame_type_code (int): The same type as its numeric wire code +/// (1, 2, 3, 5, 6, 7, 8, 9 — code 4 is reserved and never appears). +/// version (int): Frame-type-specific version from the frame header. +/// flags (int): Raw 16-bit frame flags; bit 1 is ``HASH_PRESENT`` +/// (see :attr:`has_hash`). +/// offset (int): Byte offset of the frame header within the message. +/// length (int): Whole-frame span in bytes, frame header through +/// ``ENDF``, excluding trailing alignment padding. +/// payload (bytes): The frame *content* — the 16-byte frame header and +/// the type-specific footer are excluded. For a data-object frame +/// that is the encoded payload plus the trailing CBOR descriptor; +/// for every other type it is the CBOR body. Materialised on +/// access from the source buffer, so walking frames for their +/// offsets costs no payload copies. +/// has_hash (bool): The frame's ``HASH_PRESENT`` flag — its hash slot +/// holds a real digest. +/// is_data_object (bool): ``True`` for data-object frames +/// (``NTensorFrame`` in v3). +#[pyclass(name = "Frame")] +struct PyFrame { + #[pyo3(get)] + frame_type: String, + #[pyo3(get)] + frame_type_code: u16, + #[pyo3(get)] + version: u16, + #[pyo3(get)] + flags: u16, + #[pyo3(get)] + offset: usize, + #[pyo3(get)] + length: usize, + #[pyo3(get)] + has_hash: bool, + #[pyo3(get)] + is_data_object: bool, + /// The source message, retained so :attr:`payload` can be sliced on + /// demand rather than copied for every frame the walk yields. + buf: PyBackedBytes, + payload_offset: usize, + payload_len: usize, +} + +#[pymethods] +impl PyFrame { + /// The frame's content as ``bytes`` (header and footer excluded). + #[getter] + fn payload<'py>(&self, py: Python<'py>) -> Bound<'py, PyBytes> { + PyBytes::new( + py, + &self.buf[self.payload_offset..self.payload_offset + self.payload_len], + ) + } + + fn __repr__(&self) -> String { + format!( + "Frame(frame_type='{}', offset={}, length={}, payload_len={}, has_hash={})", + self.frame_type, + self.offset, + self.length, + self.payload_len, + if self.has_hash { "True" } else { "False" }, + ) + } +} + +/// Structural description of one frame, without its payload bytes. +/// +/// `payload_offset` is an absolute offset into the source message (the +/// core reports the payload as a borrowed slice; the binding keeps the +/// span so the bytes can be produced later from the retained buffer). +struct FrameRecord { + frame_type: FrameType, + version: u16, + flags: u16, + offset: usize, + length: usize, + has_hash: bool, + payload_offset: usize, + payload_len: usize, +} + +/// Walk the frames of a **single** message. +/// +/// Direct binding of the Rust core :func:`tensogram::frames`. Yields one +/// :class:`Frame` per ``next()``, in wire order, for frame types 1–9. The +/// preamble and postamble are *not* frames and are never yielded — use +/// :func:`message_header` for the envelope. +/// +/// Args: +/// buf: The bytes of **one** message, starting at its ``TENSOGRM`` +/// preamble. For a multi-message buffer or file, slice it first:: +/// +/// for offset, length in tensogram.scan(buf): +/// for frame in tensogram.frames(buf[offset:offset + length]): +/// ... +/// +/// Passing an unsliced multi-message buffer is not an error, but the +/// walk stops at the end of the first message. +/// +/// Returns: +/// :class:`FrameIter` — an iterator of :class:`Frame` objects. +/// +/// Raises: +/// ValueError: If ``buf`` does not start with a readable preamble. A +/// preamble that parses but is followed by a malformed frame chain +/// raises from ``next()`` instead, after the frames that did parse. +/// +/// Example:: +/// +/// for frame in tensogram.frames(msg): +/// print(frame.frame_type, frame.offset, frame.length, len(frame.payload)) +#[pyfunction] +#[pyo3(name = "frames")] +fn py_frames(buf: PyBackedBytes) -> PyResult { + use tensogram_lib::wire::FRAME_HEADER_SIZE; + + // The core iterator borrows `buf`, so the structural walk (frame + // headers only — no payload is touched) runs here and the payload + // spans are resolved against the retained buffer on demand. This is + // the same shape as `objects_metadata` / `ObjectMetadataIter`. + let mut records: Vec = Vec::new(); + let mut error: Option = None; + for item in core_frames(&buf).map_err(to_py_err)? { + match item { + Ok(info) => records.push(FrameRecord { + frame_type: info.frame_type, + version: info.version, + flags: info.flags, + offset: info.offset, + length: info.length, + // The core owns the meaning of the flag bits. + has_hash: info.has_hash(), + // `FrameInfo::payload` starts right after the frame header + // and stops before the type-specific footer. + payload_offset: info.offset + FRAME_HEADER_SIZE, + payload_len: info.payload.len(), + }), + // The core iterator yields at most one error and then stops; + // it is surfaced from `__next__` in the position it occurred. + Err(e) => { + error = Some(e); + break; + } + } + } + Ok(PyFrameIter { + buf, + records, + index: 0, + error, + }) +} + +/// Iterator over the frames of a single message. +/// +/// Created by :func:`frames`. Implements ``__len__`` (frames left to +/// yield, not counting a trailing framing error). A malformed frame chain +/// raises ``ValueError`` from ``next()`` once, after the frames that did +/// parse; iteration then stops cleanly. +/// +/// Cost model: the structural walk reads frame *headers* only and never +/// touches payload bytes; each ``next()`` then materialises one +/// :class:`Frame`, whose ``payload`` is sliced out of the retained source +/// buffer on access. Walking a large message to inspect offsets and +/// lengths therefore copies nothing. +#[pyclass(name = "FrameIter")] +struct PyFrameIter { + buf: PyBackedBytes, + records: Vec, + index: usize, + /// Framing error that ended the walk, replayed once in position. + error: Option, +} + +#[pymethods] +impl PyFrameIter { + fn __iter__(slf: PyRef<'_, Self>) -> PyRef<'_, Self> { + slf + } + + fn __next__(&mut self, py: Python<'_>) -> PyResult> { + if let Some(record) = self.records.get(self.index) { + self.index += 1; + let frame = PyFrame { + frame_type: frame_type_name(record.frame_type).to_string(), + frame_type_code: record.frame_type as u16, + version: record.version, + flags: record.flags, + offset: record.offset, + length: record.length, + has_hash: record.has_hash, + is_data_object: record.frame_type.is_data_object(), + buf: self.buf.clone_ref(py), + payload_offset: record.payload_offset, + payload_len: record.payload_len, + }; + return Ok(Some(frame.into_pyobject(py)?.into_any().unbind())); + } + // `take` makes the error a one-shot: the next call ends the walk. + match self.error.take() { + Some(e) => Err(to_py_err(e)), + None => Ok(None), + } + } + + fn __len__(&self) -> usize { + self.records.len().saturating_sub(self.index) + } + + fn __repr__(&self) -> String { + let remaining = self.records.len().saturating_sub(self.index); + format!( + "FrameIter(position={}, remaining={})", + self.index, remaining + ) + } +} + +/// A message's envelope (its 24-byte preamble), decoded into typed values. +/// +/// Direct binding of the Rust core ``tensogram::MessageHeader``, returned +/// by :func:`message_header`. The eight structural predicates keep the +/// core's ``has_`` prefix and are plain read-only ``bool`` attributes, so +/// ``header.has_header_index`` reads the same in every binding. +/// +/// Attributes: +/// version (int): Wire-format version (3 in v3). +/// total_length (int): Total message length in bytes, preamble through +/// postamble. ``0`` for a streaming message whose length was never +/// back-filled. +/// flags (int): Raw preamble flag bits; prefer the predicates below. +/// has_header_metadata (bool): A ``HeaderMetadata`` frame is present +/// (random-access mode). +/// has_footer_metadata (bool): A ``FooterMetadata`` frame is present +/// (streaming mode). +/// has_header_index (bool): A ``HeaderIndex`` frame is present. +/// has_footer_index (bool): A ``FooterIndex`` frame is present. +/// has_header_hashes (bool): A ``HeaderHash`` frame is present. +/// has_footer_hashes (bool): A ``FooterHash`` frame is present. +/// has_preceder_metadata (bool): At least one ``PrecederMetadata`` +/// frame appears in the body. +/// has_hashes_present (bool): Advisory — every frame has its per-frame +/// ``HASH_PRESENT`` bit set. The per-frame :attr:`Frame.has_hash` +/// stays authoritative for any single frame. +/// +/// In a buffered (``encode``) message the flags describe the frames +/// exactly. A streaming message writes its preamble before any object, so +/// its flags are advisory: a frame that *is* present is always advertised, +/// but ``has_preceder_metadata`` may be set optimistically with no +/// ``PrecederMetadata`` frame in the body. +#[pyclass(name = "MessageHeader")] +struct PyMessageHeader { + #[pyo3(get)] + version: u16, + #[pyo3(get)] + total_length: u64, + #[pyo3(get)] + flags: u16, + #[pyo3(get)] + has_header_metadata: bool, + #[pyo3(get)] + has_footer_metadata: bool, + #[pyo3(get)] + has_header_index: bool, + #[pyo3(get)] + has_footer_index: bool, + #[pyo3(get)] + has_header_hashes: bool, + #[pyo3(get)] + has_footer_hashes: bool, + #[pyo3(get)] + has_preceder_metadata: bool, + #[pyo3(get)] + has_hashes_present: bool, +} + +#[pymethods] +impl PyMessageHeader { + fn __repr__(&self) -> String { + let mut present: Vec<&str> = Vec::new(); + for (name, set) in [ + ("header_metadata", self.has_header_metadata), + ("footer_metadata", self.has_footer_metadata), + ("header_index", self.has_header_index), + ("footer_index", self.has_footer_index), + ("header_hashes", self.has_header_hashes), + ("footer_hashes", self.has_footer_hashes), + ("preceder_metadata", self.has_preceder_metadata), + ("hashes_present", self.has_hashes_present), + ] { + if set { + present.push(name); + } + } + format!( + "MessageHeader(version={}, total_length={}, flags=0x{:04x}, present=[{}])", + self.version, + self.total_length, + self.flags, + present.join(", ") + ) + } +} + +/// Read a message's envelope without walking its frames. +/// +/// Direct binding of the Rust core :func:`tensogram::message_header`. +/// Parses only the 24-byte preamble, so it is the cheapest way to tell a +/// random-access message (metadata / index / hashes in the *header*) from a +/// streaming one (in the *footer*). +/// +/// Args: +/// buf: The bytes of **one** message, starting at its ``TENSOGRM`` +/// preamble. For a multi-message buffer, slice it with +/// :func:`scan` first; reading an unsliced buffer describes only +/// the first message. +/// +/// Returns: +/// :class:`MessageHeader`. +/// +/// Raises: +/// ValueError: If ``buf`` does not start with a readable preamble. +/// +/// Example:: +/// +/// header = tensogram.message_header(msg) +/// if header.has_header_index: +/// print("random-access message of", header.total_length, "bytes") +#[pyfunction] +#[pyo3(name = "message_header")] +fn py_message_header(buf: PyBackedBytes) -> PyResult { + let header = core_message_header(&buf).map_err(to_py_err)?; + Ok(PyMessageHeader { + version: header.version, + total_length: header.total_length, + flags: header.flags.bits(), + has_header_metadata: header.has_header_metadata(), + has_footer_metadata: header.has_footer_metadata(), + has_header_index: header.has_header_index(), + has_footer_index: header.has_footer_index(), + has_header_hashes: header.has_header_hashes(), + has_footer_hashes: header.has_footer_hashes(), + has_preceder_metadata: header.has_preceder_metadata(), + has_hashes_present: header.has_hashes_present(), + }) +} + /// Compute a hash digest over arbitrary bytes. /// /// Mirrors Rust :func:`tensogram::compute_hash`, the WASM @@ -2352,6 +2757,12 @@ impl PyStreamingEncoder { /// :meth:`write_object` (axis B only — streaming encoding /// does not have cross-object parallelism by design). /// Default ``0`` preserves the sequential path. + /// aggregate_hash: aggregate hash frame placement — ``"auto"`` + /// (default; streaming → footer), ``"none"`` or ``"footer"``. + /// ``"header"`` / ``"both"`` raise ``ValueError``: the + /// streaming header is written before any data object. + /// compression_backend: szip / zstd implementation — ``"auto"`` + /// (default), ``"ffi"``, ``"pure"``. #[new] #[pyo3( signature = ( @@ -2365,6 +2776,7 @@ impl PyStreamingEncoder { neg_inf_mask_method=None, small_mask_threshold_bytes=None, aggregate_hash=None, + compression_backend=None, ) )] #[allow(clippy::too_many_arguments)] @@ -2379,6 +2791,7 @@ impl PyStreamingEncoder { neg_inf_mask_method: Option<&str>, small_mask_threshold_bytes: Option, aggregate_hash: Option<&str>, + compression_backend: Option<&str>, ) -> PyResult { let global_meta = dict_to_global_metadata(global_meta_dict)?; let options = make_encode_options_full( @@ -2391,6 +2804,7 @@ impl PyStreamingEncoder { neg_inf_mask_method, small_mask_threshold_bytes, aggregate_hash, + compression_backend, )?; let inner = StreamingEncoder::new(std::io::Cursor::new(Vec::new()), &global_meta, &options) .map_err(to_py_err)?; @@ -3339,7 +3753,8 @@ impl PyAsyncStreamingEncoder { /// /// Writes the preamble + header metadata frame to the in-memory sink /// before the returned coroutine resolves. Accepts the same kwargs - /// as :class:`StreamingEncoder`. ``aggregate_hash="header"`` / + /// as :class:`StreamingEncoder`, including ``aggregate_hash`` and + /// ``compression_backend``. ``aggregate_hash="header"`` / /// ``"both"`` are rejected in streaming mode (the header is written /// before any data object). /// @@ -3359,6 +3774,7 @@ impl PyAsyncStreamingEncoder { neg_inf_mask_method=None, small_mask_threshold_bytes=None, aggregate_hash=None, + compression_backend=None, ) )] #[allow(clippy::too_many_arguments)] @@ -3374,6 +3790,7 @@ impl PyAsyncStreamingEncoder { neg_inf_mask_method: Option<&str>, small_mask_threshold_bytes: Option, aggregate_hash: Option<&str>, + compression_backend: Option<&str>, ) -> PyResult> { // Convert the caller's metadata + options while the GIL is held, // yielding owned `Send + 'static` values to move into the future. @@ -3388,6 +3805,7 @@ impl PyAsyncStreamingEncoder { neg_inf_mask_method, small_mask_threshold_bytes, aggregate_hash, + compression_backend, )?; pyo3_async_runtimes::tokio::future_into_py(py, async move { let enc = AsyncStreamingEncoder::new( @@ -4211,6 +4629,8 @@ fn tensogram(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(py_objects, m)?)?; m.add_function(wrap_pyfunction!(py_objects_metadata, m)?)?; m.add_function(wrap_pyfunction!(py_object_inline_hashes, m)?)?; + m.add_function(wrap_pyfunction!(py_frames, m)?)?; + m.add_function(wrap_pyfunction!(py_message_header, m)?)?; m.add_function(wrap_pyfunction!(py_compute_common, m)?)?; m.add_function(wrap_pyfunction!(py_iter_messages, m)?)?; m.add_function(wrap_pyfunction!(py_validate, m)?)?; @@ -4233,6 +4653,9 @@ fn tensogram(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_class::()?; + m.add_class::()?; + m.add_class::()?; + m.add_class::()?; m.add_class::()?; #[cfg(feature = "async")] m.add_class::()?; @@ -4282,7 +4705,9 @@ fn tensogram(m: &Bound<'_, PyModule>) -> PyResult<()> { // --------------------------------------------------------------------------- fn make_encode_options(hash: Option<&str>, threads: u32) -> PyResult { - make_encode_options_full(hash, threads, false, false, None, None, None, None, None) + make_encode_options_full( + hash, threads, false, false, None, None, None, None, None, None, + ) } /// Build an [`EncodeOptions`] from the full kwargs set exposed by the @@ -4306,6 +4731,17 @@ fn make_encode_options(hash: Option<&str>, threads: u32) -> PyResult, @@ -4317,7 +4753,9 @@ fn make_encode_options_full( neg_inf_mask_method: Option<&str>, small_mask_threshold_bytes: Option, aggregate_hash: Option<&str>, + compression_backend: Option<&str>, ) -> PyResult { + use tensogram_lib::CompressionBackend; use tensogram_lib::encode::{AggregateHashPolicy, MaskMethod}; // Python-side `hash=` semantics — distinct from the Rust core @@ -4361,6 +4799,18 @@ fn make_encode_options_full( } }; + let backend = match compression_backend { + None | Some("auto") => CompressionBackend::Auto, + Some("ffi") => CompressionBackend::Ffi, + Some("pure") => CompressionBackend::Pure, + Some(other) => { + return Err(PyValueError::new_err(format!( + "unknown compression_backend {other:?}; \ + expected one of: 'auto', 'ffi', 'pure'" + ))); + } + }; + let defaults = EncodeOptions::default(); Ok(EncodeOptions { hashing, @@ -4373,6 +4823,7 @@ fn make_encode_options_full( small_mask_threshold_bytes: small_mask_threshold_bytes .unwrap_or(defaults.small_mask_threshold_bytes), aggregate_hash: aggregate, + compression_backend: backend, ..defaults }) } diff --git a/python/tests/test_encode_knobs.py b/python/tests/test_encode_knobs.py new file mode 100644 index 00000000..1d2d9938 --- /dev/null +++ b/python/tests/test_encode_knobs.py @@ -0,0 +1,243 @@ +# (C) Copyright 2026- ECMWF and individual contributors. +# +# This software is licensed under the terms of the Apache Licence Version 2.0 +# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +# In applying this licence, ECMWF does not waive the privileges and immunities +# granted to it by virtue of its status as an intergovernmental organisation nor +# does it submit to any jurisdiction. + +"""Tests for the encode-side policy knobs ``aggregate_hash=`` and +``compression_backend=``. + +Both map onto the Rust core ``EncodeOptions`` fields of the same name and +are accepted by every encode entry point: :func:`tensogram.encode`, +:class:`tensogram.StreamingEncoder`, :meth:`tensogram.TensogramFile.append` +and :meth:`tensogram.AsyncStreamingEncoder.create`. +""" + +# pyright: basic, reportAttributeAccessIssue=false, reportMissingTypeStubs=false + +from __future__ import annotations + +import numpy as np +import pytest +import tensogram + +_DESCRIPTOR = { + "type": "ntensor", + "shape": [16], + "dtype": "float32", + "byte_order": "little", + "encoding": "none", + "filter": "none", + "compression": "none", +} +_ZSTD_DESCRIPTOR = {**_DESCRIPTOR, "compression": "zstd"} +_PAYLOAD = np.arange(16, dtype=np.float32) +_META = {"base": [{}]} + +AGGREGATE_HASH_VALUES = ["auto", "none", "header", "footer", "both"] +COMPRESSION_BACKENDS = ["auto", "ffi", "pure"] + + +def _frame_types(msg: bytes) -> list[str]: + return [f.frame_type for f in tensogram.frames(msg)] + + +# --------------------------------------------------------------------------- +# aggregate_hash= — aggregate hash frame placement +# --------------------------------------------------------------------------- + + +class TestAggregateHash: + @pytest.mark.parametrize("policy", AGGREGATE_HASH_VALUES) + def test_value_round_trips(self, policy): + msg = tensogram.encode(_META, [(_DESCRIPTOR, _PAYLOAD)], aggregate_hash=policy) + decoded = tensogram.decode(msg) + np.testing.assert_array_equal(decoded.objects[0][1], _PAYLOAD) + + @pytest.mark.parametrize( + ("policy", "header_hash", "footer_hash"), + [ + ("auto", True, False), # buffered default → header + ("none", False, False), + ("header", True, False), + ("footer", False, True), + ("both", True, True), + ], + ) + def test_placement_matches_the_policy(self, policy, header_hash, footer_hash): + types = _frame_types( + tensogram.encode(_META, [(_DESCRIPTOR, _PAYLOAD)], aggregate_hash=policy) + ) + assert ("HeaderHash" in types) is header_hash + assert ("FooterHash" in types) is footer_hash + + def test_both_places_a_hash_frame_in_the_header_and_the_footer(self): + # The closed loop: the encode knob is verified with the frame walker. + types = _frame_types( + tensogram.encode(_META, [(_DESCRIPTOR, _PAYLOAD)], aggregate_hash="both") + ) + assert "HeaderHash" in types + assert "FooterHash" in types + # ... and they sit on the correct side of the data objects. + assert types.index("HeaderHash") < types.index("NTensorFrame") + assert types.index("FooterHash") > types.index("NTensorFrame") + + @pytest.mark.parametrize("policy", AGGREGATE_HASH_VALUES) + def test_message_header_flags_track_the_placement(self, policy): + msg = tensogram.encode(_META, [(_DESCRIPTOR, _PAYLOAD)], aggregate_hash=policy) + types = _frame_types(msg) + header = tensogram.message_header(msg) + assert header.has_header_hashes == ("HeaderHash" in types) + assert header.has_footer_hashes == ("FooterHash" in types) + + def test_unknown_value_raises(self): + with pytest.raises(ValueError, match=r"unknown aggregate_hash policy"): + tensogram.encode(_META, [(_DESCRIPTOR, _PAYLOAD)], aggregate_hash="sometimes") + + def test_error_lists_the_accepted_values(self): + with pytest.raises(ValueError, match=r"'auto'.*'none'.*'header'.*'footer'.*'both'"): + tensogram.encode(_META, [(_DESCRIPTOR, _PAYLOAD)], aggregate_hash="") + + @pytest.mark.parametrize("policy", ["auto", "none", "footer"]) + def test_streaming_accepts_footer_side_policies(self, policy): + enc = tensogram.StreamingEncoder(_META, aggregate_hash=policy) + enc.write_object(_DESCRIPTOR, _PAYLOAD) + msg = enc.finish() + types = _frame_types(msg) + assert ("FooterHash" in types) is (policy != "none") + assert "HeaderHash" not in types + + @pytest.mark.parametrize("policy", ["header", "both"]) + def test_streaming_rejects_header_side_policies(self, policy): + with pytest.raises(ValueError, match=r"(?i)streaming|header"): + tensogram.StreamingEncoder(_META, aggregate_hash=policy) + + def test_file_append_accepts_the_policy(self, tmp_path): + path = str(tmp_path / "aggregate.tgm") + with tensogram.TensogramFile.create(path) as f: + f.append(_META, [(_DESCRIPTOR, _PAYLOAD)], aggregate_hash="both") + with tensogram.TensogramFile.open(path) as f: + types = _frame_types(f.read_message(0)) + assert "HeaderHash" in types + assert "FooterHash" in types + + def test_file_append_rejects_an_unknown_policy(self, tmp_path): + path = str(tmp_path / "bad.tgm") + with ( + tensogram.TensogramFile.create(path) as f, + pytest.raises(ValueError, match=r"unknown aggregate_hash policy"), + ): + f.append(_META, [(_DESCRIPTOR, _PAYLOAD)], aggregate_hash="nope") + + @pytest.mark.asyncio + async def test_async_streaming_accepts_the_policy(self): + enc = await tensogram.AsyncStreamingEncoder.create(_META, aggregate_hash="footer") + await enc.write_object(_DESCRIPTOR, _PAYLOAD) + msg = await enc.finish() + assert "FooterHash" in _frame_types(msg) + + +# --------------------------------------------------------------------------- +# compression_backend= — FFI vs pure-Rust codec selection +# --------------------------------------------------------------------------- + + +class TestCompressionBackend: + @pytest.mark.parametrize("backend", COMPRESSION_BACKENDS) + def test_value_round_trips(self, backend): + msg = tensogram.encode(_META, [(_DESCRIPTOR, _PAYLOAD)], compression_backend=backend) + decoded = tensogram.decode(msg) + np.testing.assert_array_equal(decoded.objects[0][1], _PAYLOAD) + + @pytest.mark.parametrize("backend", COMPRESSION_BACKENDS) + def test_value_round_trips_through_a_compressed_codec(self, backend): + # zstd is one of the two codecs (with szip) that has both an FFI and + # a pure-Rust implementation, so this is where the knob bites. + msg = tensogram.encode(_META, [(_ZSTD_DESCRIPTOR, _PAYLOAD)], compression_backend=backend) + decoded = tensogram.decode(msg) + descriptor, values = decoded.objects[0] + assert descriptor.compression == "zstd" + np.testing.assert_array_equal(values, _PAYLOAD) + + def test_backends_agree_on_the_decoded_values(self): + decoded = [ + tensogram.decode( + tensogram.encode(_META, [(_ZSTD_DESCRIPTOR, _PAYLOAD)], compression_backend=b) + ).objects[0][1] + for b in COMPRESSION_BACKENDS + ] + for values in decoded[1:]: + np.testing.assert_array_equal(values, decoded[0]) + + def test_default_matches_auto(self): + default = tensogram.decode(tensogram.encode(_META, [(_ZSTD_DESCRIPTOR, _PAYLOAD)])) + explicit = tensogram.decode( + tensogram.encode(_META, [(_ZSTD_DESCRIPTOR, _PAYLOAD)], compression_backend="auto") + ) + np.testing.assert_array_equal(default.objects[0][1], explicit.objects[0][1]) + + def test_unknown_value_raises(self): + with pytest.raises(ValueError, match=r"unknown compression_backend"): + tensogram.encode(_META, [(_DESCRIPTOR, _PAYLOAD)], compression_backend="cuda") + + def test_error_lists_the_accepted_values(self): + with pytest.raises(ValueError, match=r"'auto'.*'ffi'.*'pure'"): + tensogram.encode(_META, [(_DESCRIPTOR, _PAYLOAD)], compression_backend="") + + def test_value_is_case_sensitive(self): + with pytest.raises(ValueError, match=r"unknown compression_backend"): + tensogram.encode(_META, [(_DESCRIPTOR, _PAYLOAD)], compression_backend="PURE") + + @pytest.mark.parametrize("backend", COMPRESSION_BACKENDS) + def test_streaming_encoder_accepts_the_backend(self, backend): + enc = tensogram.StreamingEncoder(_META, compression_backend=backend) + enc.write_object(_ZSTD_DESCRIPTOR, _PAYLOAD) + msg = enc.finish() + np.testing.assert_array_equal(tensogram.decode(msg).objects[0][1], _PAYLOAD) + + def test_streaming_encoder_rejects_an_unknown_backend(self): + with pytest.raises(ValueError, match=r"unknown compression_backend"): + tensogram.StreamingEncoder(_META, compression_backend="turbo") + + @pytest.mark.parametrize("backend", COMPRESSION_BACKENDS) + def test_file_append_accepts_the_backend(self, backend, tmp_path): + path = str(tmp_path / f"backend_{backend}.tgm") + with tensogram.TensogramFile.create(path) as f: + f.append(_META, [(_ZSTD_DESCRIPTOR, _PAYLOAD)], compression_backend=backend) + with tensogram.TensogramFile.open(path) as f: + _meta, objects = f.decode_message(0) + np.testing.assert_array_equal(objects[0][1], _PAYLOAD) + + def test_file_append_rejects_an_unknown_backend(self, tmp_path): + path = str(tmp_path / "bad_backend.tgm") + with ( + tensogram.TensogramFile.create(path) as f, + pytest.raises(ValueError, match=r"unknown compression_backend"), + ): + f.append(_META, [(_DESCRIPTOR, _PAYLOAD)], compression_backend="turbo") + + @pytest.mark.asyncio + async def test_async_streaming_accepts_the_backend(self): + enc = await tensogram.AsyncStreamingEncoder.create(_META, compression_backend="pure") + await enc.write_object(_ZSTD_DESCRIPTOR, _PAYLOAD) + msg = await enc.finish() + np.testing.assert_array_equal(tensogram.decode(msg).objects[0][1], _PAYLOAD) + + @pytest.mark.asyncio + async def test_async_streaming_rejects_an_unknown_backend(self): + with pytest.raises(ValueError, match=r"unknown compression_backend"): + await tensogram.AsyncStreamingEncoder.create(_META, compression_backend="turbo") + + def test_both_knobs_combine(self): + msg = tensogram.encode( + _META, + [(_ZSTD_DESCRIPTOR, _PAYLOAD)], + aggregate_hash="both", + compression_backend="pure", + ) + types = _frame_types(msg) + assert "HeaderHash" in types + assert "FooterHash" in types + np.testing.assert_array_equal(tensogram.decode(msg).objects[0][1], _PAYLOAD) diff --git a/python/tests/test_frame_walk.py b/python/tests/test_frame_walk.py new file mode 100644 index 00000000..6099933a --- /dev/null +++ b/python/tests/test_frame_walk.py @@ -0,0 +1,425 @@ +# (C) Copyright 2026- ECMWF and individual contributors. +# +# This software is licensed under the terms of the Apache Licence Version 2.0 +# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +# In applying this licence, ECMWF does not waive the privileges and immunities +# granted to it by virtue of its status as an intergovernmental organisation nor +# does it submit to any jurisdiction. + +"""Tests for the structural walkers ``tensogram.frames()`` and +``tensogram.message_header()``. + +Both mirror the Rust core (``tensogram::frames`` / ``tensogram::message_header``) +and operate on **one** message: the preamble/postamble are not frames, and a +multi-message buffer must be sliced with :func:`tensogram.scan` first. +""" + +# pyright: basic, reportAttributeAccessIssue=false, reportMissingTypeStubs=false + +from __future__ import annotations + +import numpy as np +import pytest +import tensogram + +# Wire-format constants (plans/WIRE_FORMAT.md §2, §3) — spelled out here so the +# assertions below read as statements about the format, not about the binding. +PREAMBLE_SIZE = 24 +FRAME_HEADER_SIZE = 16 +COMMON_FOOTER_SIZE = 12 # [hash u64][ENDF] +DATA_OBJECT_FOOTER_SIZE = 20 # [cbor_offset u64][hash u64][ENDF] + +# Frame-type name → numeric code (plans/WIRE_FORMAT.md §4). Type 4 is +# reserved (obsolete v2 tensor frame) and never appears. +FRAME_TYPE_CODES = { + "HeaderMetadata": 1, + "HeaderIndex": 2, + "HeaderHash": 3, + "FooterHash": 5, + "FooterIndex": 6, + "FooterMetadata": 7, + "PrecederMetadata": 8, + "NTensorFrame": 9, +} + +_DESCRIPTOR = { + "type": "ntensor", + "shape": [4], + "dtype": "float32", + "byte_order": "little", + "encoding": "none", + "filter": "none", + "compression": "none", +} + + +def _payload(fill: float = 1.0) -> np.ndarray: + return np.full(4, fill, dtype=np.float32) + + +def _buffered(n_objects: int = 2, **kwargs) -> bytes: + """Buffered (random-access) encode: metadata/index/hash live in the header.""" + pairs = [(_DESCRIPTOR, _payload(float(i))) for i in range(n_objects)] + return tensogram.encode({"base": [{} for _ in range(n_objects)]}, pairs, **kwargs) + + +def _streamed(n_objects: int = 1, *, preceders: bool = False, **kwargs) -> bytes: + """Streaming encode: index/hash land in the footer.""" + enc = tensogram.StreamingEncoder({"base": [{}]}, **kwargs) + for i in range(n_objects): + if preceders: + enc.write_preceder({"index": i}) + enc.write_object(_DESCRIPTOR, _payload(float(i))) + return enc.finish() + + +def _types(msg: bytes) -> list[str]: + return [f.frame_type for f in tensogram.frames(msg)] + + +def _walk_until_error(walker) -> tuple[list[str], Exception | None]: + """Drain a frame walker, returning what it yielded before it raised.""" + seen: list[str] = [] + try: + for frame in walker: + seen.append(frame.frame_type) + except ValueError as exc: + return seen, exc + return seen, None + + +# --------------------------------------------------------------------------- +# frames() — the frame walker +# --------------------------------------------------------------------------- + + +class TestFrames: + def test_buffered_frame_type_sequence(self): + # Buffered mode with default options: metadata, index and the + # aggregate hash frame in the header, then one frame per object. + assert _types(_buffered(2)) == [ + "HeaderMetadata", + "HeaderIndex", + "HeaderHash", + "NTensorFrame", + "NTensorFrame", + ] + + def test_one_data_object_frame_per_encoded_object(self): + for n in (1, 3, 5): + assert _types(_buffered(n)).count("NTensorFrame") == n + + def test_frame_type_code_pairs_with_the_name(self): + for f in tensogram.frames(_buffered(2)): + assert f.frame_type_code == FRAME_TYPE_CODES[f.frame_type] + + def test_is_data_object_only_for_ntensor_frames(self): + for f in tensogram.frames(_buffered(2)): + assert f.is_data_object == (f.frame_type == "NTensorFrame") + + def test_preamble_is_not_a_frame(self): + first = next(tensogram.frames(_buffered(1))) + assert first.offset == PREAMBLE_SIZE + + def test_postamble_is_not_a_frame(self): + msg = _buffered(1) + last = list(tensogram.frames(msg))[-1] + # The walk stops before the 24-byte postamble. + assert last.offset + last.length <= len(msg) - 24 + + def test_offsets_and_lengths_are_in_bounds_and_do_not_overlap(self): + msg = _buffered(3) + prev_end = PREAMBLE_SIZE + for f in tensogram.frames(msg): + assert f.offset >= prev_end + assert f.length >= FRAME_HEADER_SIZE + assert f.offset + f.length <= len(msg) + prev_end = f.offset + f.length + + def test_frame_span_starts_with_the_frame_magic_and_ends_with_endf(self): + msg = _buffered(2) + for f in tensogram.frames(msg): + span = msg[f.offset : f.offset + f.length] + assert span[:2] == b"FR" + assert span[-4:] == b"ENDF" + + def test_payload_is_the_content_slice_of_the_frame(self): + msg = _buffered(2) + for f in tensogram.frames(msg): + span = msg[f.offset : f.offset + f.length] + assert isinstance(f.payload, bytes) + assert f.payload == span[FRAME_HEADER_SIZE : FRAME_HEADER_SIZE + len(f.payload)] + + def test_payload_excludes_the_header_and_the_type_specific_footer(self): + for f in tensogram.frames(_buffered(2)): + footer = DATA_OBJECT_FOOTER_SIZE if f.is_data_object else COMMON_FOOTER_SIZE + assert len(f.payload) == f.length - FRAME_HEADER_SIZE - footer + + def test_uncompressed_object_payload_starts_with_the_raw_values(self): + # encoding/filter/compression are all "none", so the data-object + # frame content begins with the little-endian input bytes (the CBOR + # descriptor follows). + msg = _buffered(1) + obj = next(f for f in tensogram.frames(msg) if f.is_data_object) + assert obj.payload[:16] == _payload(0.0).tobytes() + + def test_version_and_flags_are_ints(self): + for f in tensogram.frames(_buffered(1)): + assert isinstance(f.version, int) + assert isinstance(f.flags, int) + assert f.version == 1 + + def test_has_hash_set_for_every_frame_by_default(self): + assert all(f.has_hash for f in tensogram.frames(_buffered(2))) + + def test_has_hash_clear_when_hashing_is_disabled(self): + assert not any(f.has_hash for f in tensogram.frames(_buffered(2, hash=None))) + + def test_has_hash_mirrors_the_per_frame_flag_bit(self): + for f in tensogram.frames(_buffered(1)): + assert f.has_hash == bool(f.flags & (1 << 1)) + + def test_is_an_iterator(self): + it = tensogram.frames(_buffered(1)) + assert iter(it) is it + + def test_iteration_is_lazy_and_resumable(self): + msg = _buffered(3) + total = len(_types(msg)) + it = tensogram.frames(msg) + first = next(it) + assert first.frame_type == "HeaderMetadata" + # The remainder is still there — pulling one frame does not consume + # the rest of the walk. + assert len(list(it)) == total - 1 + + def test_stops_cleanly(self): + it = tensogram.frames(_buffered(1)) + for _ in range(len(_types(_buffered(1)))): + next(it) + with pytest.raises(StopIteration): + next(it) + + def test_len_reports_the_frames_left_to_yield(self): + msg = _buffered(2) + it = tensogram.frames(msg) + assert len(it) == len(_types(msg)) + + def test_len_decrements(self): + it = tensogram.frames(_buffered(2)) + before = len(it) + next(it) + assert len(it) == before - 1 + + def test_len_is_zero_when_exhausted(self): + it = tensogram.frames(_buffered(1)) + list(it) + assert len(it) == 0 + + def test_frame_repr(self): + f = next(tensogram.frames(_buffered(1))) + text = repr(f) + assert "Frame(" in text + assert "HeaderMetadata" in text + assert "offset=" in text + + def test_iterator_repr(self): + assert "FrameIter(" in repr(tensogram.frames(_buffered(1))) + + def test_streaming_frame_type_sequence(self): + # Streaming mode writes the metadata header first, then objects, and + # defers metadata/hash/index to the footer. + assert _types(_streamed(2)) == [ + "HeaderMetadata", + "NTensorFrame", + "NTensorFrame", + "FooterMetadata", + "FooterHash", + "FooterIndex", + ] + + def test_streaming_preceder_metadata_frames_are_walked(self): + types = _types(_streamed(2, preceders=True)) + assert types.count("PrecederMetadata") == 2 + # Each preceder immediately precedes its data-object frame. + for i, name in enumerate(types): + if name == "PrecederMetadata": + assert types[i + 1] == "NTensorFrame" + + def test_streaming_message_without_backfilled_length_still_walks(self): + # finish() leaves total_length = 0; the walk must still stop at the + # postamble rather than mistaking it for a frame. + msg = _streamed(1) + assert tensogram.message_header(msg).total_length == 0 + assert _types(msg)[-1] == "FooterIndex" + + def test_garbage_buffer_raises(self): + with pytest.raises(ValueError, match=r"(?i)framing|magic|too short"): + tensogram.frames(b"not a tensogram message at all") + + def test_empty_buffer_raises(self): + with pytest.raises(ValueError, match=r"(?i)framing|magic|too short"): + tensogram.frames(b"") + + def test_truncated_frame_chain_raises_during_iteration_then_stops(self): + msg = _buffered(2) + last = list(tensogram.frames(msg))[-1] + truncated = msg[: last.offset + 8] # cut inside the final frame + it = tensogram.frames(truncated) # the preamble still parses + with pytest.raises(ValueError, match=r"(?i)framing|truncated|does not fit"): + list(it) + # Iteration stops after the error is surfaced. + with pytest.raises(StopIteration): + next(it) + + def test_frames_before_the_truncation_are_still_yielded(self): + msg = _buffered(2) + last = list(tensogram.frames(msg))[-1] + seen, error = _walk_until_error(tensogram.frames(msg[: last.offset + 8])) + assert seen == ["HeaderMetadata", "HeaderIndex", "HeaderHash", "NTensorFrame"] + assert error is not None + assert "framing" in str(error).lower() + + def test_multi_message_buffer_is_sliced_with_scan(self): + buf = _buffered(1) + _buffered(3) + spans = tensogram.scan(buf) + assert len(spans) == 2 + second_off, second_len = spans[1] + second = _types(buf[second_off : second_off + second_len]) + assert second.count("NTensorFrame") == 3 + # Offsets are relative to the start of the message that was passed in. + assert next(tensogram.frames(buf[second_off : second_off + second_len])).offset == ( + PREAMBLE_SIZE + ) + + def test_unsliced_multi_message_buffer_walks_only_the_first_message(self): + buf = _buffered(1) + _buffered(3) + assert _types(buf).count("NTensorFrame") == 1 + + def test_accepts_bytearray(self): + msg = _buffered(1) + types = [f.frame_type for f in tensogram.frames(bytearray(msg))] + assert types == _types(msg) + + def test_message_without_objects_walks_to_its_metadata_frame(self): + # No objects → the encoder builds neither an index nor an aggregate + # hash frame, so the walk is a single metadata frame. + msg = tensogram.encode({"base": []}, []) + assert _types(msg) == ["HeaderMetadata"] + header = tensogram.message_header(msg) + assert header.has_header_metadata + assert not header.has_header_index + assert not header.has_header_hashes + + def test_a_frame_outlives_the_iterator_that_produced_it(self): + # Each Frame keeps its own reference to the source buffer, so its + # payload stays readable after the walker is gone. + walker = tensogram.frames(_buffered(1)) + frame = next(walker) + del walker + assert len(frame.payload) == frame.length - FRAME_HEADER_SIZE - COMMON_FOOTER_SIZE + + +# --------------------------------------------------------------------------- +# message_header() — the typed preamble +# --------------------------------------------------------------------------- + + +class TestMessageHeader: + def test_version_and_total_length(self): + msg = _buffered(2) + header = tensogram.message_header(msg) + assert header.version == tensogram.WIRE_VERSION + assert header.total_length == len(msg) + + def test_flags_raw_bits(self): + header = tensogram.message_header(_buffered(1)) + assert isinstance(header.flags, int) + assert header.flags & (1 << 0) # HEADER_METADATA + + def test_buffered_flags_match_the_frames_exactly(self): + msg = _buffered(2) + header = tensogram.message_header(msg) + present = set(_types(msg)) + assert header.has_header_metadata == ("HeaderMetadata" in present) + assert header.has_footer_metadata == ("FooterMetadata" in present) + assert header.has_header_index == ("HeaderIndex" in present) + assert header.has_footer_index == ("FooterIndex" in present) + assert header.has_header_hashes == ("HeaderHash" in present) + assert header.has_footer_hashes == ("FooterHash" in present) + assert header.has_preceder_metadata == ("PrecederMetadata" in present) + + def test_buffered_predicates_are_bools(self): + header = tensogram.message_header(_buffered(1)) + for name in ( + "has_header_metadata", + "has_footer_metadata", + "has_header_index", + "has_footer_index", + "has_header_hashes", + "has_footer_hashes", + "has_preceder_metadata", + "has_hashes_present", + ): + assert isinstance(getattr(header, name), bool), name + + def test_hashes_present_tracks_the_hashing_option(self): + assert tensogram.message_header(_buffered(1)).has_hashes_present is True + assert tensogram.message_header(_buffered(1, hash=None)).has_hashes_present is False + + def test_streaming_flags_never_understate_the_frames_present(self): + # The streaming preamble is written before any object, so it is + # advisory: PRECEDER_METADATA is set optimistically. The invariant + # that always holds is "frame present ⇒ flag set". + msg = _streamed(1) + header = tensogram.message_header(msg) + present = set(_types(msg)) + implications = { + "HeaderMetadata": header.has_header_metadata, + "FooterMetadata": header.has_footer_metadata, + "HeaderIndex": header.has_header_index, + "FooterIndex": header.has_footer_index, + "HeaderHash": header.has_header_hashes, + "FooterHash": header.has_footer_hashes, + "PrecederMetadata": header.has_preceder_metadata, + } + for name, flag in implications.items(): + assert (name not in present) or flag, name + assert "FooterIndex" in present + + def test_streaming_total_length_is_zero_until_backfilled(self): + enc = tensogram.StreamingEncoder({"base": [{}]}) + enc.write_object(_DESCRIPTOR, _payload()) + msg = enc.finish() + assert tensogram.message_header(msg).total_length == 0 + + def test_backfilled_streaming_total_length_matches(self): + enc = tensogram.StreamingEncoder({"base": [{}]}) + enc.write_object(_DESCRIPTOR, _payload()) + msg = enc.finish_backfilled() + assert tensogram.message_header(msg).total_length == len(msg) + + def test_repr(self): + text = repr(tensogram.message_header(_buffered(1))) + assert "MessageHeader(" in text + assert "version=" in text + assert "total_length=" in text + + def test_multi_message_buffer_is_sliced_with_scan(self): + buf = _buffered(1) + _buffered(3) + off, length = tensogram.scan(buf)[1] + assert tensogram.message_header(buf[off : off + length]).total_length == length + # Reading the unsliced buffer describes the *first* message. + assert tensogram.message_header(buf).total_length == tensogram.scan(buf)[0][1] + + def test_garbage_buffer_raises(self): + with pytest.raises(ValueError, match=r"(?i)framing|magic|too short"): + tensogram.message_header(b"not a tensogram message at all") + + def test_empty_buffer_raises(self): + with pytest.raises(ValueError, match=r"(?i)framing|magic|too short"): + tensogram.message_header(b"") + + def test_accepts_bytearray(self): + msg = _buffered(1) + assert tensogram.message_header(bytearray(msg)).total_length == len(msg) diff --git a/rust/tensogram-ffi/Cargo.toml b/rust/tensogram-ffi/Cargo.toml index 6f399fb0..b9cd5f75 100644 --- a/rust/tensogram-ffi/Cargo.toml +++ b/rust/tensogram-ffi/Cargo.toml @@ -31,6 +31,12 @@ async = [ # Enables the remote-aware async surface (open_remote etc.). Implies # `async`. async-remote = ["async", "tensogram/remote"] +# Enables the *synchronous* remote surface (tgm_is_remote_url, +# tgm_file_open_remote). Deliberately independent of `async`: a C caller +# reading an S3 / GCS / Azure / HTTP `.tgm` through the ordinary blocking +# file API should not have to adopt the async ABI. `async-remote` does not +# imply it either — each surface is switched on explicitly. +remote = ["tensogram/remote"] [dependencies] tensogram = { path = "../tensogram", version = "=0.24.0" } diff --git a/rust/tensogram-ffi/cbindgen.toml b/rust/tensogram-ffi/cbindgen.toml index f67ce27e..3cbc32a0 100644 --- a/rust/tensogram-ffi/cbindgen.toml +++ b/rust/tensogram-ffi/cbindgen.toml @@ -27,6 +27,17 @@ usize_is_size_t = true "TgmValueType" = "tgm_value_type" # Mask-kind enum: no _t suffix so prefix_with_name produces TGM_MASK_KIND_NAN "TgmMaskKind" = "tgm_mask_kind" +# Frame-type enum: no _t suffix so prefix_with_name produces TGM_FRAME_TYPE_NTENSOR +"TgmFrameType" = "tgm_frame_type" +# Element-type enum: no _t suffix so prefix_with_name produces TGM_DTYPE_FLOAT32 +"TgmDtype" = "tgm_dtype" +# Byte-order enum: no _t suffix so prefix_with_name produces TGM_BYTE_ORDER_LITTLE +"TgmByteOrder" = "tgm_byte_order" +# Aggregate-hash-policy enum: prefix_with_name produces TGM_AGGREGATE_HASH_POLICY_BOTH +"TgmAggregateHashPolicy" = "tgm_aggregate_hash_policy" +# Compression-backend enum: prefix_with_name produces TGM_COMPRESSION_BACKEND_PURE +"TgmCompressionBackend" = "tgm_compression_backend" +"TgmFrameIter" = "tgm_frame_iter_t" "TgmValue" = "tgm_value_t" "TgmBytes" = "tgm_bytes_t" "TgmMessage" = "tgm_message_t" diff --git a/rust/tensogram-ffi/src/lib.rs b/rust/tensogram-ffi/src/lib.rs index 7211d8f6..14ca0d7c 100644 --- a/rust/tensogram-ffi/src/lib.rs +++ b/rust/tensogram-ffi/src/lib.rs @@ -59,8 +59,8 @@ use tensogram::{ DataObjectDescriptor, DecodeOptions, DecodedMaskSet, EncodeOptions, GlobalMetadata, MetaType, MetaValue, RESERVED_KEY, ScanOptions, StreamingEncoder, TensogramError, TensogramFile, compute_common, decode, decode_metadata, decode_object, decode_range, decode_with_masks, - encode, encode_pre_encoded, parse_hash_name, scan, scan_file, scan_file_with_options, - scan_with_options, verify_canonical_cbor, + encode, encode_pre_encoded, message_header as core_message_header, parse_hash_name, scan, + scan_file, scan_file_with_options, scan_with_options, verify_canonical_cbor, }; // --------------------------------------------------------------------------- @@ -174,6 +174,19 @@ fn set_last_error(msg: &str) { LAST_ERROR_OBJECT_INDEX.with(|c| c.set(-1)); } +/// Discard the thread-local last error, so [`tgm_last_error`] reports NULL. +/// +/// Needed wherever "nothing went wrong" has to be observable *through the +/// error channel* rather than the return value — e.g. [`tgm_frame_iter_next`] +/// returns `false` both at a clean end and on a malformed frame, and callers +/// tell the two apart by whether a last error is set. +fn clear_last_error() { + LAST_ERROR.with(|cell| { + *cell.borrow_mut() = None; + }); + LAST_ERROR_OBJECT_INDEX.with(|c| c.set(-1)); +} + /// Returns a pointer to the last error message, or NULL if no error. /// The pointer is valid until the next FFI call on the same thread. #[unsafe(no_mangle)] @@ -286,6 +299,158 @@ unsafe fn apply_mask_options( Ok(()) } +/// Where to place the aggregate hash frame (cbindgen: +/// `tgm_aggregate_hash_policy`, variants `TGM_AGGREGATE_HASH_POLICY_*`). +/// +/// Mirrors [`tensogram::AggregateHashPolicy`]. `AUTO` is the zero value, so a +/// zero-initialised [`TgmEncodeOptions`] asks the encoder to choose +/// (`HEADER` when buffering, `FOOTER` when streaming) — the same default the +/// Rust API has. +/// +/// `HEADER` and `BOTH` are **buffered-mode only**: a streaming encoder writes +/// its header before any data object exists, so the per-object hashes are not +/// yet known. [`tgm_streaming_encoder_create_with_encode_options`] rejects +/// them with `TGM_ERROR_ENCODING` and an explanatory +/// [`tgm_last_error`] message. +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TgmAggregateHashPolicy { + /// Encoder picks: buffered → header, streaming → footer. + Auto = 0, + /// Emit no aggregate hash frame. Per-frame inline hash slots are + /// unaffected. + None = 1, + /// Emit a `TGM_FRAME_TYPE_HEADER_HASH` frame. Buffered mode only. + Header = 2, + /// Emit a `TGM_FRAME_TYPE_FOOTER_HASH` frame. Valid in both modes. + Footer = 3, + /// Emit both a header and a footer hash frame carrying identical hash + /// lists. Buffered mode only. + Both = 4, +} + +impl From for tensogram::AggregateHashPolicy { + fn from(p: TgmAggregateHashPolicy) -> Self { + match p { + TgmAggregateHashPolicy::Auto => tensogram::AggregateHashPolicy::Auto, + TgmAggregateHashPolicy::None => tensogram::AggregateHashPolicy::None, + TgmAggregateHashPolicy::Header => tensogram::AggregateHashPolicy::Header, + TgmAggregateHashPolicy::Footer => tensogram::AggregateHashPolicy::Footer, + TgmAggregateHashPolicy::Both => tensogram::AggregateHashPolicy::Both, + } + } +} + +/// Which codec implementation to use where both are compiled in (cbindgen: +/// `tgm_compression_backend`, variants `TGM_COMPRESSION_BACKEND_*`). +/// +/// Mirrors [`tensogram::CompressionBackend`]. `AUTO` is the zero value: +/// consult `TENSOGRAM_COMPRESSION_BACKEND`, else the platform default (FFI on +/// native, pure-Rust on wasm32). `FFI` and `PURE` are explicit overrides that +/// always win over the environment. +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TgmCompressionBackend { + /// Consult the environment, then the platform default. + Auto = 0, + /// Use the C FFI codecs (libaec for szip, libzstd for zstd). + Ffi = 1, + /// Use the pure-Rust codecs (tensogram-szip, ruzstd). + Pure = 2, +} + +impl From for tensogram::CompressionBackend { + fn from(b: TgmCompressionBackend) -> Self { + match b { + TgmCompressionBackend::Auto => tensogram::CompressionBackend::Auto, + TgmCompressionBackend::Ffi => tensogram::CompressionBackend::Ffi, + TgmCompressionBackend::Pure => tensogram::CompressionBackend::Pure, + } + } +} + +/// The full encode-side option set (cbindgen: `TgmEncodeOptions`). +/// +/// Supersedes [`TgmEncodeMaskOptions`], which stays for source compatibility: +/// it carries the same six mask fields **plus** the three knobs that +/// previously had no C surface at all — the hash algorithm, where the +/// aggregate hash frame goes, and which codec backend to use. +/// +/// Pass `NULL` to any `*_with_encode_options` entry point for the library +/// defaults; a zero-initialised struct means the same thing, because every +/// field's zero value *is* the default: +/// +/// | field | `NULL` / zero | meaning | +/// |---|---|---| +/// | `hash` | NULL | no hashing (the FFI convention: name it to get it) | +/// | `aggregate_hash` | `TGM_AGGREGATE_HASH_POLICY_AUTO` | encoder picks the placement | +/// | `compression_backend` | `TGM_COMPRESSION_BACKEND_AUTO` | env, then platform default | +/// | `allow_nan` / `allow_inf` | `false` | non-finite input is a hard error | +/// | `*_mask_method` | NULL | the library default (`"roaring"`) | +/// | `small_mask_threshold_bytes` | negative | the library default (128) | +/// +/// `hash` is `"xxh3"` (v3's only algorithm), `"none"`, or NULL; anything else +/// is [`TgmError::InvalidArg`]. Each `*_mask_method` is one of `"none"`, +/// `"rle"`, `"roaring"`, `"lz4"`, `"zstd"`, `"blosc2"`. +/// `small_mask_threshold_bytes` is the byte count below which mask blobs are +/// stored raw regardless of the requested method; `0` disables that +/// auto-fallback, negative values select the library default. +#[repr(C)] +pub struct TgmEncodeOptions { + /// Hash algorithm name (`"xxh3"`, `"none"`) or NULL for no hashing. + pub hash: *const c_char, + /// Where to write the aggregate hash frame. Ignored when `hash` is NULL + /// or `"none"` — there is nothing to aggregate. + pub aggregate_hash: TgmAggregateHashPolicy, + /// Which codec implementation to prefer for szip / zstd. + pub compression_backend: TgmCompressionBackend, + /// Substitute NaN with `0.0` and record a bitmask companion frame. + pub allow_nan: bool, + /// Substitute `±Inf` with `0.0` and record per-sign bitmask companions. + pub allow_inf: bool, + /// Compression method for the NaN mask, or NULL for the default. + pub nan_mask_method: *const c_char, + /// Compression method for the `+Inf` mask, or NULL for the default. + pub pos_inf_mask_method: *const c_char, + /// Compression method for the `-Inf` mask, or NULL for the default. + pub neg_inf_mask_method: *const c_char, + /// Raw-storage threshold for mask blobs; `0` disables, negative = default. + pub small_mask_threshold_bytes: isize, +} + +/// Apply the optional [`TgmEncodeOptions`] pointer to an [`EncodeOptions`]. +/// `NULL` is a no-op, leaving the caller's defaults in place. Returns an +/// error message (routed to [`set_last_error`] by the caller) when the hash +/// algorithm or a mask method name is invalid UTF-8 or unknown. +/// +/// # Safety +/// +/// `opts` must either be `NULL` or point to a valid `TgmEncodeOptions` whose +/// string fields are NULL or NUL-terminated. +unsafe fn apply_encode_options( + encode_opts: &mut EncodeOptions, + opts: *const TgmEncodeOptions, +) -> Result<(), String> { + let Some(opts) = (unsafe { opts.as_ref() }) else { + return Ok(()); + }; + encode_opts.hashing = parse_hash_algo(opts.hash).map_err(|(_code, msg)| msg)?; + encode_opts.aggregate_hash = opts.aggregate_hash.into(); + encode_opts.compression_backend = opts.compression_backend.into(); + encode_opts.allow_nan = opts.allow_nan; + encode_opts.allow_inf = opts.allow_inf; + encode_opts.nan_mask_method = + unsafe { parse_mask_method_cstr(opts.nan_mask_method, MaskMethod::default())? }; + encode_opts.pos_inf_mask_method = + unsafe { parse_mask_method_cstr(opts.pos_inf_mask_method, MaskMethod::default())? }; + encode_opts.neg_inf_mask_method = + unsafe { parse_mask_method_cstr(opts.neg_inf_mask_method, MaskMethod::default())? }; + if opts.small_mask_threshold_bytes >= 0 { + encode_opts.small_mask_threshold_bytes = opts.small_mask_threshold_bytes as usize; + } + Ok(()) +} + /// Decode-side companion to [`TgmEncodeMaskOptions`]. Pass a pointer /// to opt out of canonical NaN / Inf restoration. Pass `NULL` for /// the default `restore_non_finite = true`. @@ -1030,6 +1195,90 @@ pub extern "C" fn tgm_encode_with_options( } } +/// Encode with the full [`TgmEncodeOptions`] set. +/// +/// Like [`tgm_encode_with_options`], but the option struct also carries the +/// hash algorithm, the aggregate-hash placement and the compression backend — +/// so there is no separate `hash_algo` argument here; put the algorithm name +/// in `options->hash`. `NULL` options behave exactly like [`tgm_encode`] with +/// a NULL `hash_algo`: no hashing, `AUTO` placement, `AUTO` backend, +/// non-finite input rejected. +/// +/// On success returns `TGM_ERROR_OK` and fills `out` with the encoded bytes, +/// which the caller frees with [`tgm_bytes_free`]. +#[unsafe(no_mangle)] +pub extern "C" fn tgm_encode_with_encode_options( + metadata_json: *const c_char, + data_ptrs: *const *const u8, + data_lens: *const usize, + num_objects: usize, + threads: u32, + options: *const TgmEncodeOptions, + out: *mut TgmBytes, +) -> TgmError { + if metadata_json.is_null() || out.is_null() { + set_last_error("null argument"); + return TgmError::InvalidArg; + } + + let json_str = match unsafe { CStr::from_ptr(metadata_json) }.to_str() { + Ok(s) => s, + Err(e) => { + set_last_error(&format!("invalid UTF-8 in metadata_json: {e}")); + return TgmError::InvalidArg; + } + }; + + // The hash algorithm now travels inside `options`; pass NULL to the + // shared parser and let `apply_encode_options` set `hashing`. + let mut parsed = match unsafe { + parse_encode_args( + json_str, + data_ptrs, + data_lens, + num_objects, + ptr::null(), + threads, + ) + } { + Ok(p) => p, + Err((code, msg)) => { + set_last_error(&msg); + return code; + } + }; + if let Err(msg) = unsafe { apply_encode_options(&mut parsed.options, options) } { + set_last_error(&msg); + return TgmError::InvalidArg; + } + + let pairs: Vec<(&DataObjectDescriptor, &[u8])> = parsed + .descriptors + .iter() + .zip(parsed.data_slices.iter()) + .map(|(d, s)| (d, *s)) + .collect(); + + match encode(&parsed.global_metadata, &pairs, &parsed.options) { + Ok(bytes) => { + let mut bytes = bytes.into_boxed_slice().into_vec(); + let result = TgmBytes { + data: bytes.as_mut_ptr(), + len: bytes.len(), + }; + std::mem::forget(bytes); + unsafe { + *out = result; + } + TgmError::Ok + } + Err(e) => { + set_last_error(&e.to_string()); + to_error_code(&e) + } + } +} + /// Decode with explicit NaN / Inf restoration options. /// /// Like [`tgm_decode`] but takes a [`TgmDecodeMaskOptions`] pointer @@ -1180,6 +1429,87 @@ pub extern "C" fn tgm_streaming_encoder_create_with_options( } } +/// Streaming-encoder constructor taking the full [`TgmEncodeOptions`] set. +/// +/// Like [`tgm_streaming_encoder_create_with_options`], but the option struct +/// also carries the hash algorithm, the aggregate-hash placement and the +/// compression backend, so there is no separate `hash_algo` argument. `NULL` +/// options behave like [`tgm_streaming_encoder_create`] with a NULL +/// `hash_algo`. +/// +/// `TGM_AGGREGATE_HASH_POLICY_HEADER` and `..._BOTH` are rejected here with +/// `TGM_ERROR_ENCODING`: a streaming writer emits its header before any data +/// object exists, so the per-object hashes are not yet known. Use `AUTO` +/// (which resolves to the footer when streaming) or `FOOTER`. +#[unsafe(no_mangle)] +pub extern "C" fn tgm_streaming_encoder_create_with_encode_options( + path: *const c_char, + metadata_json: *const c_char, + threads: u32, + options: *const TgmEncodeOptions, + out: *mut *mut TgmStreamingEncoder, +) -> TgmError { + if path.is_null() || metadata_json.is_null() || out.is_null() { + set_last_error("null argument"); + return TgmError::InvalidArg; + } + let path_str = match unsafe { CStr::from_ptr(path) }.to_str() { + Ok(s) => s, + Err(e) => { + set_last_error(&format!("invalid UTF-8 in path: {e}")); + return TgmError::InvalidArg; + } + }; + let json_str = match unsafe { CStr::from_ptr(metadata_json) }.to_str() { + Ok(s) => s, + Err(e) => { + set_last_error(&format!("invalid UTF-8 in metadata_json: {e}")); + return TgmError::InvalidArg; + } + }; + let global_metadata = match parse_streaming_metadata_json(json_str) { + Ok(m) => m, + Err(e) => { + set_last_error(&e); + return TgmError::Metadata; + } + }; + // Parse the option *names* before creating the file, so a typo'd hash + // algorithm or mask method cannot truncate an existing one. Whether the + // aggregate-hash placement is legal in streaming mode stays core's call + // (`StreamingEncoder::new`), so that rule has one home. + let mut encode_options = EncodeOptions { + threads, + hashing: false, + ..Default::default() + }; + if let Err(msg) = unsafe { apply_encode_options(&mut encode_options, options) } { + set_last_error(&msg); + return TgmError::InvalidArg; + } + let file = match std::fs::File::create(path_str) { + Ok(f) => f, + Err(e) => { + set_last_error(&e.to_string()); + return TgmError::Io; + } + }; + let writer = std::io::BufWriter::new(file); + match StreamingEncoder::new(writer, &global_metadata, &encode_options) { + Ok(enc) => { + let handle = Box::new(TgmStreamingEncoder { inner: Some(enc) }); + unsafe { + *out = Box::into_raw(handle); + } + TgmError::Ok + } + Err(e) => { + set_last_error(&e.to_string()); + to_error_code(&e) + } + } +} + /// Append a message to a file with explicit NaN / Inf mask-companion options. /// /// Like [`tgm_file_append`] but takes a [`TgmEncodeMaskOptions`] @@ -1243,6 +1573,72 @@ pub extern "C" fn tgm_file_append_with_options( } } +/// Append a message to a file with the full [`TgmEncodeOptions`] set. +/// +/// Like [`tgm_file_append_with_options`], but the option struct also carries +/// the hash algorithm, the aggregate-hash placement and the compression +/// backend, so there is no separate `hash_algo` argument. `NULL` options +/// behave like [`tgm_file_append`] with a NULL `hash_algo`. +#[unsafe(no_mangle)] +#[allow(clippy::too_many_arguments)] +pub extern "C" fn tgm_file_append_with_encode_options( + file: *mut TgmFile, + metadata_json: *const c_char, + data_ptrs: *const *const u8, + data_lens: *const usize, + num_objects: usize, + threads: u32, + options: *const TgmEncodeOptions, +) -> TgmError { + if file.is_null() || metadata_json.is_null() { + set_last_error("null argument"); + return TgmError::InvalidArg; + } + let json_str = match unsafe { CStr::from_ptr(metadata_json) }.to_str() { + Ok(s) => s, + Err(e) => { + set_last_error(&format!("invalid UTF-8 in metadata_json: {e}")); + return TgmError::InvalidArg; + } + }; + // The hash algorithm travels inside `options`; see + // [`tgm_encode_with_encode_options`]. + let mut parsed = match unsafe { + parse_encode_args( + json_str, + data_ptrs, + data_lens, + num_objects, + ptr::null(), + threads, + ) + } { + Ok(p) => p, + Err((code, msg)) => { + set_last_error(&msg); + return code; + } + }; + if let Err(msg) = unsafe { apply_encode_options(&mut parsed.options, options) } { + set_last_error(&msg); + return TgmError::InvalidArg; + } + let pairs: Vec<(&DataObjectDescriptor, &[u8])> = parsed + .descriptors + .iter() + .zip(parsed.data_slices.iter()) + .map(|(d, s)| (d, *s)) + .collect(); + let f = unsafe { &mut (*file).file }; + match f.append(&parsed.global_metadata, &pairs, &parsed.options) { + Ok(()) => TgmError::Ok, + Err(e) => { + set_last_error(&e.to_string()); + to_error_code(&e) + } + } +} + /// Encode a Tensogram message from JSON metadata and pre-encoded payload bytes. /// /// Like `tgm_encode`, but each `data_ptrs[i]` slice must already be encoded @@ -2040,6 +2436,84 @@ pub extern "C" fn tgm_object_strides(msg: *const TgmMessage, index: usize) -> *c } } +/// An element type (cbindgen: `tgm_dtype`, variants `TGM_DTYPE_*`). +/// +/// Mirrors [`tensogram::Dtype`] one-for-one, numbered densely from `0` in core +/// declaration order. The wire format stores the dtype as a **string** (see +/// `plans/WIRE_FORMAT.md` §6.1), so these codes are an FFI convenience, not a +/// wire value — but they are part of the C ABI and therefore frozen. +/// [`tgm_object_dtype`] returns the same information as a string; this enum is +/// what you want in a `switch`. +/// +/// The `impl From` below is an exhaustive match, so a new core variant +/// fails the build here rather than silently mapping to something wrong. +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TgmDtype { + Float16 = 0, + Bfloat16 = 1, + Float32 = 2, + Float64 = 3, + Complex64 = 4, + Complex128 = 5, + Int8 = 6, + Int16 = 7, + Int32 = 8, + Int64 = 9, + Uint8 = 10, + Uint16 = 11, + Uint32 = 12, + Uint64 = 13, + /// Sub-byte packed bitmask; element width is 1 bit, so the byte width of + /// [`tensogram::Dtype::Bitmask`] is reported as 0. + Bitmask = 14, +} + +impl From for TgmDtype { + fn from(d: tensogram::Dtype) -> Self { + match d { + tensogram::Dtype::Float16 => TgmDtype::Float16, + tensogram::Dtype::Bfloat16 => TgmDtype::Bfloat16, + tensogram::Dtype::Float32 => TgmDtype::Float32, + tensogram::Dtype::Float64 => TgmDtype::Float64, + tensogram::Dtype::Complex64 => TgmDtype::Complex64, + tensogram::Dtype::Complex128 => TgmDtype::Complex128, + tensogram::Dtype::Int8 => TgmDtype::Int8, + tensogram::Dtype::Int16 => TgmDtype::Int16, + tensogram::Dtype::Int32 => TgmDtype::Int32, + tensogram::Dtype::Int64 => TgmDtype::Int64, + tensogram::Dtype::Uint8 => TgmDtype::Uint8, + tensogram::Dtype::Uint16 => TgmDtype::Uint16, + tensogram::Dtype::Uint32 => TgmDtype::Uint32, + tensogram::Dtype::Uint64 => TgmDtype::Uint64, + tensogram::Dtype::Bitmask => TgmDtype::Bitmask, + } + } +} + +/// A payload's byte order (cbindgen: `tgm_byte_order`, variants +/// `TGM_BYTE_ORDER_*`). +/// +/// Mirrors [`tensogram::ByteOrder`]. Like [`TgmDtype`] these codes are an FFI +/// convenience — the wire stores `"little"` / `"big"` as text — and are frozen +/// as part of the C ABI. [`tgm_object_byte_order`] returns the same +/// information as a string. +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TgmByteOrder { + Little = 0, + Big = 1, +} + +impl From for TgmByteOrder { + fn from(b: tensogram::ByteOrder) -> Self { + match b { + tensogram::ByteOrder::Little => TgmByteOrder::Little, + tensogram::ByteOrder::Big => TgmByteOrder::Big, + } + } +} + /// Returns the dtype as a null-terminated string (e.g. "float32"). /// The pointer is valid until the message is freed. #[unsafe(no_mangle)] @@ -2052,6 +2526,27 @@ pub extern "C" fn tgm_object_dtype(msg: *const TgmMessage, index: usize) -> *con } } +/// Returns the object's dtype as a [`tgm_dtype`](TgmDtype) code — the typed +/// companion to [`tgm_object_dtype`], for callers that want to `switch` +/// instead of `strcmp`. +/// +/// A NULL `msg` or an out-of-range `index` records the reason in +/// [`tgm_last_error`] and returns the zero-valued variant +/// (`TGM_DTYPE_FLOAT16`); an enum return has no spare code to signal +/// failure. To bounds-check unambiguously, compare `index` against +/// [`tgm_message_num_objects`], or call [`tgm_object_dtype`], which returns +/// NULL for exactly the same inputs. +#[unsafe(no_mangle)] +pub extern "C" fn tgm_object_dtype_enum(msg: *const TgmMessage, index: usize) -> TgmDtype { + match unsafe { as_msg(msg) }.and_then(|m| m.objects.get(index)) { + Some((desc, _)) => desc.dtype.into(), + None => { + set_last_error("null message handle or object index out of range"); + TgmDtype::Float16 + } + } +} + /// Returns a pointer to the decoded payload bytes for a decoded object. /// `decoded_index` is the index into the decoded objects array (0 for the /// first decoded object, regardless of the original object index). @@ -2167,6 +2662,24 @@ pub extern "C" fn tgm_object_byte_order(msg: *const TgmMessage, index: usize) -> } } +/// Returns the object's byte order as a [`tgm_byte_order`](TgmByteOrder) code +/// — the typed companion to [`tgm_object_byte_order`]. +/// +/// A NULL `msg` or an out-of-range `index` records the reason in +/// [`tgm_last_error`] and returns the zero-valued variant +/// (`TGM_BYTE_ORDER_LITTLE`); see [`tgm_object_dtype_enum`] for how to +/// bounds-check unambiguously. +#[unsafe(no_mangle)] +pub extern "C" fn tgm_object_byte_order_enum(msg: *const TgmMessage, index: usize) -> TgmByteOrder { + match unsafe { as_msg(msg) }.and_then(|m| m.objects.get(index)) { + Some((desc, _)) => desc.byte_order.into(), + None => { + set_last_error("null message handle or object index out of range"); + TgmByteOrder::Little + } + } +} + /// Returns the filter string (e.g. "none", "shuffle"). Valid until message freed. #[unsafe(no_mangle)] pub extern "C" fn tgm_object_filter(msg: *const TgmMessage, index: usize) -> *const c_char { @@ -3418,7 +3931,206 @@ pub extern "C" fn tgm_file_close(file: *mut TgmFile) { } // --------------------------------------------------------------------------- -// Metadata key lookup helpers +// Synchronous remote access (Cargo feature `remote`) +// --------------------------------------------------------------------------- +// +// A remote `.tgm` is opened into an ordinary `tgm_file_t`, so every existing +// reader entry point (`tgm_file_message_count`, `tgm_file_read_message`, +// `tgm_file_decode_message`, `tgm_file_iter_*`, …) works unchanged on an S3 / +// GCS / Azure / HTTP source. This is the blocking counterpart to +// `tgm_async_file_open_remote`; a C caller no longer has to adopt the whole +// async ABI just to read a remote file. +// +// Both entry points are exported unconditionally so consumers linking the +// cdylib never see an undefined symbol. Without the `remote` feature they +// report — through `tgm_last_error()` — that the build lacks the feature, +// exactly like the async surface does. + +/// `true` when `source` is a URL this build can open remotely. +/// +/// Mirrors [`tensogram::is_remote_url`]: the recognised schemes are `s3`, +/// `s3a`, `gs`, `az`, `azure`, `http` and `https`, compared +/// case-insensitively. Plain paths and `file://` URLs are **not** remote — +/// they belong to the local backend ([`tgm_file_open`]). +/// +/// Returns `false` — with the reason in [`tgm_last_error`] — for a NULL or +/// non-UTF-8 `source`, and for **every** input when this build was compiled +/// without the `remote` Cargo feature (such a build genuinely cannot open any +/// remote URL, so "not remote for me" is the honest answer; the error message +/// says how to fix it). +#[unsafe(no_mangle)] +pub extern "C" fn tgm_is_remote_url(source: *const c_char) -> bool { + if source.is_null() { + set_last_error("null argument"); + return false; + } + let source_str = match unsafe { CStr::from_ptr(source) }.to_str() { + Ok(s) => s, + Err(e) => { + set_last_error(&format!("invalid UTF-8 in source: {e}")); + return false; + } + }; + + #[cfg(not(feature = "remote"))] + { + let _ = source_str; + set_last_error( + "tgm_is_remote_url: this build of tensogram-ffi was compiled without \ + the `remote` Cargo feature, so no URL can be opened remotely; \ + rebuild with --features=remote to enable S3/GCS/Azure/HTTP support", + ); + false + } + + #[cfg(feature = "remote")] + { + tensogram::is_remote_url(source_str) + } +} + +/// Reader-side scan-walker options for [`tgm_file_open_remote`]. Mirrors +/// [`tensogram::RemoteScanOptions`] as a flat, C-visible POD so callers can +/// build it on the stack. +/// +/// Pass a `NULL` `TgmRemoteScanOptions*` to use the library defaults +/// (`bidirectional = true`). +#[repr(C)] +#[derive(Clone, Copy)] +pub struct TgmRemoteScanOptions { + /// Enable the meet-in-the-middle (bidirectional) remote walk, which + /// pairs forward preamble fetches with backward postamble fetches and + /// roughly halves wall-clock layout discovery on real networks. `false` + /// forces a forward-only walk. + pub bidirectional: bool, +} + +/// Marshal the parallel `keys` / `values` C arrays into the `BTreeMap` the +/// core remote backend takes. `n_options == 0` ignores both pointers (they +/// may be NULL); otherwise both arrays — and every entry in them — must be +/// non-NULL, valid UTF-8 strings. +/// +/// # Safety +/// +/// When `n_options > 0`, `keys` and `values` must each point to at least +/// `n_options` readable `*const c_char`, each of which is NULL or a +/// NUL-terminated string. +unsafe fn collect_storage_options( + keys: *const *const c_char, + values: *const *const c_char, + n_options: usize, +) -> Result, String> { + let mut storage = BTreeMap::new(); + if n_options == 0 { + return Ok(storage); + } + if keys.is_null() || values.is_null() { + return Err(format!( + "storage option arrays must both be non-NULL when n_options ({n_options}) > 0" + )); + } + for i in 0..n_options { + let kp = unsafe { *keys.add(i) }; + let vp = unsafe { *values.add(i) }; + if kp.is_null() || vp.is_null() { + return Err(format!("storage option {i} has a NULL key or value")); + } + let k = unsafe { CStr::from_ptr(kp) } + .to_str() + .map_err(|e| format!("invalid UTF-8 in storage key {i}: {e}"))?; + let v = unsafe { CStr::from_ptr(vp) } + .to_str() + .map_err(|e| format!("invalid UTF-8 in storage value {i}: {e}"))?; + storage.insert(k.to_string(), v.to_string()); + } + Ok(storage) +} + +/// Open a remote `.tgm` (S3 / GCS / Azure / HTTP) for **synchronous** reading. +/// +/// The blocking counterpart to `tgm_async_file_open_remote`: on success `*out` +/// receives an ordinary [`tgm_file_t`](TgmFile), so the whole existing file +/// API — [`tgm_file_message_count`], [`tgm_file_read_message`], +/// [`tgm_file_decode_message`], [`tgm_file_iter_create`], … — works unchanged +/// against the remote source. Close it with [`tgm_file_close`] as usual. +/// +/// `keys` / `values` are parallel arrays of `n_options` backend storage +/// options (credentials, region, endpoint, …) forwarded verbatim to the +/// object-store backend; pass `0` / NULL for none. `opts` configures the scan +/// walker; NULL selects the library defaults (`bidirectional = true`). +/// +/// Argument validation (NULL `source` / `out`, malformed option arrays) runs +/// **before** the feature check, so every build answers +/// `TGM_ERROR_INVALID_ARG` to the same mistakes. Remote failures — an +/// unparseable URL, a missing object, a rejected storage option, transport +/// errors — map to `TGM_ERROR_REMOTE` with the detail in [`tgm_last_error`]. +/// +/// Always exported so consumers linking the cdylib never see an undefined +/// symbol; a build without the `remote` Cargo feature returns +/// `TGM_ERROR_REMOTE` and explains how to enable it. +#[unsafe(no_mangle)] +pub extern "C" fn tgm_file_open_remote( + source: *const c_char, + keys: *const *const c_char, + values: *const *const c_char, + n_options: usize, + opts: *const TgmRemoteScanOptions, + out: *mut *mut TgmFile, +) -> TgmError { + if source.is_null() || out.is_null() { + set_last_error("null argument"); + return TgmError::InvalidArg; + } + let source_str = match unsafe { CStr::from_ptr(source) }.to_str() { + Ok(s) => s, + Err(e) => { + set_last_error(&format!("invalid UTF-8 in source: {e}")); + return TgmError::InvalidArg; + } + }; + let storage = match unsafe { collect_storage_options(keys, values, n_options) } { + Ok(m) => m, + Err(msg) => { + set_last_error(&msg); + return TgmError::InvalidArg; + } + }; + + #[cfg(not(feature = "remote"))] + { + let _ = (source_str, storage, opts); + set_last_error( + "tgm_file_open_remote: this build of tensogram-ffi was compiled without \ + the `remote` Cargo feature; rebuild with --features=remote to enable \ + S3/GCS/Azure/HTTP support", + ); + TgmError::Remote + } + + #[cfg(feature = "remote")] + { + let scan_opts = unsafe { opts.as_ref() }.map(|o| tensogram::RemoteScanOptions { + bidirectional: o.bidirectional, + }); + match TensogramFile::open_remote(source_str, &storage, scan_opts) { + Ok(file) => { + let path_string = CString::new(source_str).unwrap_or_default(); + let handle = Box::new(TgmFile { file, path_string }); + unsafe { + *out = Box::into_raw(handle); + } + TgmError::Ok + } + Err(e) => { + set_last_error(&e.to_string()); + to_error_code(&e) + } + } + } +} + +// --------------------------------------------------------------------------- +// Metadata key lookup helpers // --------------------------------------------------------------------------- // // The dot-path walkers (first-match across `base[i]`, `_extra_` fallback, @@ -4008,6 +4720,322 @@ pub extern "C" fn tgm_object_iter_free(iter: *mut TgmObjectIter) { } } +// --------------------------------------------------------------------------- +// Frame walker + message header +// --------------------------------------------------------------------------- + +/// A frame's type identifier (cbindgen: `tgm_frame_type`, variants +/// `TGM_FRAME_TYPE_*`). +/// +/// Mirrors [`tensogram::FrameType`] value-for-value — these numbers are the +/// wire's frame-type field (see `plans/WIRE_FORMAT.md` §2.2), not an FFI +/// invention. **Type 4 is reserved** (it held the obsolete v2 data-object +/// layout) and therefore has no variant, which is why the sequence skips from +/// 3 to 5. `fortran/test/check_frame_type_enum.sh` guards the generated header +/// against the core enum so the two can never drift. +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TgmFrameType { + /// CBOR global metadata, written in the header (random-access mode). + HeaderMetadata = 1, + /// Object index, written in the header (random-access mode). + HeaderIndex = 2, + /// Aggregate hash frame, written in the header. + HeaderHash = 3, + /// Aggregate hash frame, written in the footer. + FooterHash = 5, + /// Object index, written in the footer (streaming mode). + FooterIndex = 6, + /// CBOR global metadata, written in the footer (streaming mode). + FooterMetadata = 7, + /// Per-object metadata frame immediately preceding a data-object frame. + PrecederMetadata = 8, + /// N-dimensional tensor data-object frame — the only data-object type in + /// v3, and the only frame type with a 20-byte footer. + Ntensor = 9, +} + +impl From for TgmFrameType { + fn from(t: tensogram::FrameType) -> Self { + match t { + tensogram::FrameType::HeaderMetadata => TgmFrameType::HeaderMetadata, + tensogram::FrameType::HeaderIndex => TgmFrameType::HeaderIndex, + tensogram::FrameType::HeaderHash => TgmFrameType::HeaderHash, + tensogram::FrameType::FooterHash => TgmFrameType::FooterHash, + tensogram::FrameType::FooterIndex => TgmFrameType::FooterIndex, + tensogram::FrameType::FooterMetadata => TgmFrameType::FooterMetadata, + tensogram::FrameType::PrecederMetadata => TgmFrameType::PrecederMetadata, + tensogram::FrameType::NTensorFrame => TgmFrameType::Ntensor, + } + } +} + +/// One frame's structural description plus a **borrowed** view of its content. +/// +/// Filled by [`tgm_frame_iter_next`] into caller-provided storage. +/// +/// # Lifetime — read this before storing a `TgmFrame` +/// +/// `payload` points **into the `msg` buffer the caller passed to +/// [`tgm_frame_iter_create`]** — it is a view, never a copy, and there is +/// nothing to free. It stays valid for exactly as long as `msg` does, +/// independently of any later `tgm_frame_iter_next` call and of +/// `tgm_frame_iter_free`. Copy the bytes out if you need them to outlive the +/// message buffer. +#[repr(C)] +#[derive(Debug, Clone, Copy)] +pub struct TgmFrame { + /// Which kind of frame this is. + pub frame_type: TgmFrameType, + /// Frame-type-specific version field from the frame header. + pub version: u16, + /// Raw 16-bit frame flags; bit 1 is `HASH_PRESENT` (see + /// [`tgm_frame_has_hash`]). + pub flags: u16, + /// Byte offset of the frame header, relative to the start of `msg`. + pub offset: usize, + /// Whole-frame span in bytes: frame header through `ENDF`, excluding any + /// alignment padding that follows. + pub length: usize, + /// Borrowed content bytes: everything between the 16-byte frame header and + /// the type-specific footer (20 bytes for `TGM_FRAME_TYPE_NTENSOR`, 12 for + /// every other type). Points into the caller's `msg` — never freed. + pub payload: *const u8, + /// Length of `payload` in bytes. + pub payload_len: usize, +} + +/// Opaque lazy cursor over one message's frames (cbindgen: +/// `tgm_frame_iter_t`). Created by [`tgm_frame_iter_create`], advanced with +/// [`tgm_frame_iter_next`], released with [`tgm_frame_iter_free`]. +/// +/// # Soundness invariant +/// +/// The handle stores the caller's message slice with a `'static` lifetime, +/// which is a controlled fiction — exactly like the metadata value cursor's +/// arena (see [`store_value`]). The C contract that makes it sound is stated +/// on [`tgm_frame_iter_create`]: **`msg` must outlive the iterator**. Nothing +/// inside the handle owns or copies the message bytes, so freeing the handle +/// touches only the cursor; the caller's buffer is never read after +/// [`tgm_frame_iter_free`] returns. +pub struct TgmFrameIter { + /// The borrowed core cursor. Reconstructing it per call is impossible + /// (its position is private), so the borrow is carried in the handle. + inner: tensogram::FrameIter<'static>, +} + +/// Start a lazy walk over the frames of one message. +/// +/// `msg` must point at the start of a message (the `TENSOGRM` preamble magic) +/// — typically a slice obtained from `tgm_scan`. Only the type 1–9 `FR` frames +/// are yielded; the preamble and postamble are not frames, use +/// [`tgm_message_header`] for the envelope. +/// +/// Returns NULL — with the reason in `tgm_last_error` — if `msg` is NULL or +/// the preamble does not parse (truncated buffer, wrong magic, unsupported +/// version). Free the returned handle with [`tgm_frame_iter_free`]. +/// +/// # Lifetime contract +/// +/// The iterator **borrows `msg`; `msg` must outlive the iterator** and must +/// not be moved, reallocated, or mutated while the iterator lives. Each frame +/// written by [`tgm_frame_iter_next`] carries a `payload` pointer **into +/// `msg`**, which remains valid for as long as `msg` lives — later `next` +/// calls and `tgm_frame_iter_free` do not invalidate it. +/// +/// Binds [`tensogram::frames`]. +#[unsafe(no_mangle)] +pub extern "C" fn tgm_frame_iter_create(msg: *const u8, msg_len: usize) -> *mut TgmFrameIter { + if msg.is_null() { + set_last_error("null argument"); + return ptr::null_mut(); + } + // SAFETY: `msg` is non-null and the caller guarantees `msg_len` readable + // bytes. Naming the borrow `'static` is the lifetime fiction documented on + // `TgmFrameIter`: the C contract requires `msg` to outlive the handle, and + // the handle never exposes the slice beyond the borrowed `payload` + // pointers it hands back, which carry the same contract. + let data: &'static [u8] = unsafe { slice::from_raw_parts(msg, msg_len) }; + match tensogram::frames(data) { + Ok(inner) => Box::into_raw(Box::new(TgmFrameIter { inner })), + Err(e) => { + set_last_error(&e.to_string()); + ptr::null_mut() + } + } +} + +/// Advance the frame cursor, filling `*out` with the next frame. +/// +/// Returns `true` when a frame was written. Returns `false` in three cases, +/// which `tgm_last_error` tells apart: +/// +/// - **clean end** — every frame has been yielded; the last error is +/// *cleared*, so `tgm_last_error()` returns NULL; +/// - **malformed frame** — the frame chain is truncated or inconsistent; the +/// reason is recorded in `tgm_last_error()` and iteration stops for good; +/// - **invalid argument** — `it` or `out` is NULL; nothing is written. +/// +/// Calling this again after any `false` is safe and keeps returning `false`. +/// +/// `out->payload` borrows the caller's message buffer — see the lifetime +/// contract on [`tgm_frame_iter_create`]. +#[unsafe(no_mangle)] +pub extern "C" fn tgm_frame_iter_next(it: *mut TgmFrameIter, out: *mut TgmFrame) -> bool { + if it.is_null() || out.is_null() { + set_last_error("null argument"); + return false; + } + let iter = unsafe { &mut *it }; + match iter.inner.next() { + Some(Ok(info)) => { + let frame = TgmFrame { + frame_type: info.frame_type.into(), + version: info.version, + flags: info.flags, + offset: info.offset, + length: info.length, + payload: info.payload.as_ptr(), + payload_len: info.payload.len(), + }; + unsafe { + *out = frame; + } + true + } + Some(Err(e)) => { + set_last_error(&e.to_string()); + false + } + None => { + // A clean end is not a failure: clear the thread-local error so + // callers can use `tgm_last_error() != NULL` to detect the + // malformed-frame case above. + clear_last_error(); + false + } + } +} + +/// Free a frame cursor. Releases only the cursor — the caller's message +/// buffer and every `payload` pointer handed out by [`tgm_frame_iter_next`] +/// are untouched and stay valid. NULL is a no-op. +#[unsafe(no_mangle)] +pub extern "C" fn tgm_frame_iter_free(it: *mut TgmFrameIter) { + if !it.is_null() { + unsafe { + drop(Box::from_raw(it)); + } + } +} + +/// `true` if this frame's `HASH_PRESENT` flag is set, i.e. its inline hash +/// slot holds a meaningful digest (see `plans/WIRE_FORMAT.md` §2.5). +/// +/// Convenience over `frame->flags` bit 1, and the authoritative answer for a +/// single frame — `TgmMessageHeader::has_hashes_present` is only an advisory +/// message-wide summary. Returns `false` for a NULL `frame`. +/// +/// Binds [`tensogram::FrameInfo::has_hash`]. +#[unsafe(no_mangle)] +pub extern "C" fn tgm_frame_has_hash(frame: *const TgmFrame) -> bool { + match unsafe { frame.as_ref() } { + Some(f) => f.flags & tensogram::wire::FrameFlags::HASH_PRESENT != 0, + None => { + set_last_error("null argument"); + false + } + } +} + +/// A message's envelope (the 24-byte preamble) as a flat C POD. +/// +/// Filled by [`tgm_message_header`]. The eight `has_*` members are the +/// preamble's structural flags decoded into named booleans, so callers never +/// touch a raw bitset. Together they say whether a message is *random-access* +/// (metadata / index / hashes in the **header**) or *streaming* (in the +/// **footer**) without reading a single frame. +/// +/// `total_length` is the whole-message byte count, preamble through +/// postamble. It is `0` in a streaming message whose length was never +/// back-filled — not an error, just "unknown at write time". +#[repr(C)] +pub struct TgmMessageHeader { + /// Wire-format version (`TGM_WIRE_VERSION` for messages this build writes). + pub version: u16, + /// Total message length in bytes, or `0` if a streaming writer never + /// back-filled it. + pub total_length: u64, + /// A `HeaderMetadata` frame is present (random-access mode). + pub has_header_metadata: bool, + /// A `FooterMetadata` frame is present (streaming mode). + pub has_footer_metadata: bool, + /// A `HeaderIndex` frame is present. + pub has_header_index: bool, + /// A `FooterIndex` frame is present. + pub has_footer_index: bool, + /// A `HeaderHash` frame is present. + pub has_header_hashes: bool, + /// A `FooterHash` frame is present. + pub has_footer_hashes: bool, + /// At least one `PrecederMetadata` frame appears in the body. Advisory in + /// streaming mode: the encoder sets it before it knows whether any + /// preceder will be written, so `true` does not guarantee a frame. + pub has_preceder_metadata: bool, + /// Advisory: every frame in this message has its per-frame `HASH_PRESENT` + /// bit set. For any single frame, `tgm_frame_has_hash` stays authoritative. + pub has_hashes_present: bool, +} + +/// Read a message's envelope without walking its frames. +/// +/// `msg` must point at the start of a message (the `TENSOGRM` preamble magic) +/// — typically a slice obtained from `tgm_scan`. On success writes the decoded +/// preamble to `*out` and returns `TGM_ERROR_OK`. +/// +/// Returns `TGM_ERROR_INVALID_ARG` if `msg` or `out` is NULL, or the mapped +/// error code if the preamble does not parse (message truncated, wrong magic, +/// unsupported version); the reason is available from `tgm_last_error` and +/// `*out` is left untouched. +/// +/// Binds [`tensogram::message_header`]. +#[unsafe(no_mangle)] +pub extern "C" fn tgm_message_header( + msg: *const u8, + msg_len: usize, + out: *mut TgmMessageHeader, +) -> TgmError { + if msg.is_null() || out.is_null() { + set_last_error("null argument"); + return TgmError::InvalidArg; + } + let data = unsafe { slice::from_raw_parts(msg, msg_len) }; + match core_message_header(data) { + Ok(h) => { + let header = TgmMessageHeader { + version: h.version, + total_length: h.total_length, + has_header_metadata: h.has_header_metadata(), + has_footer_metadata: h.has_footer_metadata(), + has_header_index: h.has_header_index(), + has_footer_index: h.has_footer_index(), + has_header_hashes: h.has_header_hashes(), + has_footer_hashes: h.has_footer_hashes(), + has_preceder_metadata: h.has_preceder_metadata(), + has_hashes_present: h.has_hashes_present(), + }; + unsafe { + *out = header; + } + TgmError::Ok + } + Err(e) => { + set_last_error(&e.to_string()); + to_error_code(&e) + } + } +} + // --------------------------------------------------------------------------- // Error code to string // --------------------------------------------------------------------------- @@ -8964,6 +9992,1684 @@ mod tests { let mars = tgm_metadata_get(hp, CString::new("mars").unwrap().as_ptr()); assert!(tgm_value_map_get(mars, ptr::null()).is_null()); } + + // ===================================================================== + // Frame walker + message header (tgm_frame_iter_* / tgm_message_header) + // ===================================================================== + + /// One trivial float32 tensor: descriptor plus its raw little-endian bytes. + fn frame_walk_object() -> (DataObjectDescriptor, Vec) { + let desc = DataObjectDescriptor { + obj_type: "ntensor".to_string(), + ndim: 1, + shape: vec![4], + strides: vec![1], + dtype: tensogram::Dtype::Float32, + byte_order: tensogram::ByteOrder::native(), + encoding: "none".to_string(), + filter: "none".to_string(), + compression: "none".to_string(), + params: BTreeMap::new(), + masks: None, + }; + let data: Vec = [1.0f32, 2.0, 3.0, 4.0] + .iter() + .flat_map(|v| v.to_ne_bytes()) + .collect(); + (desc, data) + } + + /// A buffered (random-access) message: metadata / index / hashes all live + /// in the header, so every preamble flag is an exact statement about the + /// frames that follow. + fn frame_walk_buffered_message(n_objects: usize) -> Vec { + let (desc, data) = frame_walk_object(); + let objects: Vec<(&DataObjectDescriptor, &[u8])> = + (0..n_objects).map(|_| (&desc, data.as_slice())).collect(); + encode( + &GlobalMetadata::default(), + &objects, + &EncodeOptions::default(), + ) + .expect("buffered encode") + } + + /// A streaming message: index / hashes land in the footer, `total_length` + /// stays 0 (never back-filled by the non-seeking `finish`), and + /// `PRECEDER_METADATA` is set advisorily even with no preceder frame. + fn frame_walk_streamed_message() -> Vec { + let (desc, data) = frame_walk_object(); + let mut enc = StreamingEncoder::new( + std::io::Cursor::new(Vec::new()), + &GlobalMetadata::default(), + &EncodeOptions::default(), + ) + .expect("streaming encoder"); + enc.write_object(&desc, &data).expect("write object"); + enc.finish().expect("finish").into_inner() + } + + /// A `TgmMessageHeader` with every field cleared, so a test can tell an + /// untouched out-param from one the callee filled in. + fn blank_message_header() -> super::TgmMessageHeader { + super::TgmMessageHeader { + version: 0, + total_length: 0, + has_header_metadata: false, + has_footer_metadata: false, + has_header_index: false, + has_footer_index: false, + has_header_hashes: false, + has_footer_hashes: false, + has_preceder_metadata: false, + has_hashes_present: false, + } + } + + #[test] + fn ffi_message_header_reads_a_buffered_preamble() { + let msg = frame_walk_buffered_message(2); + let mut h = blank_message_header(); + let err = super::tgm_message_header(msg.as_ptr(), msg.len(), &mut h); + assert!(matches!(err, super::TgmError::Ok)); + assert_eq!(h.version, super::TGM_WIRE_VERSION); + assert_eq!(h.total_length, msg.len() as u64); + // Random-access layout: metadata + index in the header, nothing in the + // footer, and no preceder frames were written. + assert!(h.has_header_metadata); + assert!(h.has_header_index); + assert!(!h.has_footer_metadata); + assert!(!h.has_footer_index); + assert!(!h.has_preceder_metadata); + assert!(h.has_hashes_present, "default encode hashes every frame"); + } + + #[test] + fn ffi_message_header_reads_a_streaming_preamble() { + let msg = frame_walk_streamed_message(); + let mut h = blank_message_header(); + let err = super::tgm_message_header(msg.as_ptr(), msg.len(), &mut h); + assert!(matches!(err, super::TgmError::Ok)); + assert_eq!(h.version, super::TGM_WIRE_VERSION); + // Streaming mode cannot know the length up front and never seeks back. + assert_eq!(h.total_length, 0); + assert!(h.has_header_metadata); + assert!(h.has_footer_metadata); + assert!(h.has_footer_index); + } + + #[test] + fn ffi_message_header_null_safety() { + let msg = frame_walk_buffered_message(1); + let mut h = blank_message_header(); + assert!(matches!( + super::tgm_message_header(ptr::null(), 0, &mut h), + super::TgmError::InvalidArg + )); + assert!(matches!( + super::tgm_message_header(msg.as_ptr(), msg.len(), ptr::null_mut()), + super::TgmError::InvalidArg + )); + // A rejected call must not have written anything. + assert_eq!(h.version, 0); + } + + #[test] + fn ffi_message_header_rejects_a_buffer_that_is_not_a_message() { + let junk = b"not a tensogram message at all!!!!!!!!!!"; + let mut h = blank_message_header(); + let err = super::tgm_message_header(junk.as_ptr(), junk.len(), &mut h); + assert!(!matches!(err, super::TgmError::Ok)); + assert!(!super::tgm_last_error().is_null(), "reason is reported"); + assert_eq!(h.version, 0, "a failed parse leaves the out-param alone"); + + // A truncated preamble is rejected the same way. + let msg = frame_walk_buffered_message(1); + let err = super::tgm_message_header(msg.as_ptr(), 8, &mut h); + assert!(!matches!(err, super::TgmError::Ok)); + } + + /// A `TgmFrame` with every field cleared, so a test can tell an untouched + /// out-param from one the callee filled in (`payload` stays NULL). + fn blank_frame() -> super::TgmFrame { + super::TgmFrame { + frame_type: super::TgmFrameType::HeaderMetadata, + version: 0, + flags: 0, + offset: 0, + length: 0, + payload: ptr::null(), + payload_len: 0, + } + } + + /// Walk `msg` to exhaustion through the C API and collect every frame. + /// Asserts the clean-end contract: an exhausted walk records no error. + fn ffi_collect_frames(msg: &[u8]) -> Vec { + let it = super::tgm_frame_iter_create(msg.as_ptr(), msg.len()); + assert!(!it.is_null(), "iterator create failed"); + let mut frames = Vec::new(); + let mut f = blank_frame(); + while super::tgm_frame_iter_next(it, &mut f) { + frames.push(f); + } + assert!( + super::tgm_last_error().is_null(), + "a clean end must not look like a malformed frame" + ); + super::tgm_frame_iter_free(it); + frames + } + + #[test] + fn ffi_frame_type_discriminants_match_the_core_wire_types() { + // The C enum carries the wire's frame-type value; drift would silently + // mislabel frames. `fortran/test/check_frame_type_enum.sh` guards the + // generated header against the same source of truth. + for (c, core) in [ + ( + super::TgmFrameType::HeaderMetadata, + tensogram::FrameType::HeaderMetadata, + ), + ( + super::TgmFrameType::HeaderIndex, + tensogram::FrameType::HeaderIndex, + ), + ( + super::TgmFrameType::HeaderHash, + tensogram::FrameType::HeaderHash, + ), + ( + super::TgmFrameType::FooterHash, + tensogram::FrameType::FooterHash, + ), + ( + super::TgmFrameType::FooterIndex, + tensogram::FrameType::FooterIndex, + ), + ( + super::TgmFrameType::FooterMetadata, + tensogram::FrameType::FooterMetadata, + ), + ( + super::TgmFrameType::PrecederMetadata, + tensogram::FrameType::PrecederMetadata, + ), + ( + super::TgmFrameType::Ntensor, + tensogram::FrameType::NTensorFrame, + ), + ] { + assert_eq!(c as u16, core as u16, "{c:?} must mirror {core:?}"); + } + assert_eq!(super::TgmFrameType::HeaderMetadata as u16, 1); + assert_eq!(super::TgmFrameType::Ntensor as u16, 9); + // 4 is reserved (the obsolete v2 data-object frame) — no variant here. + assert!(tensogram::FrameType::from_u16(4).is_err()); + } + + #[test] + fn ffi_frame_iter_mirrors_the_core_walk_exactly() { + let msg = frame_walk_buffered_message(2); + let ffi = ffi_collect_frames(&msg); + let core: Vec<_> = tensogram::frames(&msg) + .expect("core frames") + .map(|r| r.expect("core frame")) + .collect(); + assert_eq!(ffi.len(), core.len(), "same number of frames"); + for (f, c) in ffi.iter().zip(&core) { + assert_eq!(f.frame_type as u16, c.frame_type as u16); + assert_eq!(f.version, c.version); + assert_eq!(f.flags, c.flags); + assert_eq!(f.offset, c.offset); + assert_eq!(f.length, c.length); + assert_eq!(f.payload_len, c.payload.len()); + assert_eq!(f.payload, c.payload.as_ptr(), "payload borrows the caller"); + } + } + + #[test] + fn ffi_frame_iter_yields_the_expected_frame_sequence() { + let msg = frame_walk_buffered_message(3); + let fs = ffi_collect_frames(&msg); + assert!(!fs.is_empty(), "expected frames"); + assert_eq!( + fs[0].frame_type, + super::TgmFrameType::HeaderMetadata, + "a random-access message opens with its metadata frame" + ); + let data_objects = fs + .iter() + .filter(|f| f.frame_type == super::TgmFrameType::Ntensor) + .count(); + assert_eq!(data_objects, 3, "one data-object frame per encoded object"); + // The preamble and the postamble are NOT frames. + assert!(fs[0].offset >= tensogram::wire::PREAMBLE_SIZE); + let last = fs.last().expect("at least one frame"); + assert!(last.offset + last.length <= msg.len() - tensogram::wire::POSTAMBLE_SIZE); + } + + #[test] + fn ffi_frame_offsets_and_lengths_tile_the_message_in_bounds() { + let msg = frame_walk_buffered_message(2); + let mut prev_end = tensogram::wire::PREAMBLE_SIZE; + for f in ffi_collect_frames(&msg) { + assert!(f.offset >= prev_end, "frames must not overlap"); + assert!(f.offset + f.length <= msg.len(), "frame stays in bounds"); + assert!(f.length >= tensogram::wire::FRAME_HEADER_SIZE); + // The whole-frame span always ends on the ENDF sentinel. + let end = f.offset + f.length; + assert_eq!(&msg[end - 4..end], b"ENDF", "length spans the footer"); + prev_end = end; + } + assert!( + prev_end > tensogram::wire::PREAMBLE_SIZE, + "walked at least one frame" + ); + } + + #[test] + fn ffi_frame_payload_excludes_the_frame_header_and_footer() { + let msg = frame_walk_buffered_message(1); + let fs = ffi_collect_frames(&msg); + assert!(!fs.is_empty()); + for f in fs { + let core_type = + tensogram::FrameType::from_u16(f.frame_type as u16).expect("known frame type"); + // 20 bytes for a data-object frame, 12 for every other type. + let footer = tensogram::wire::footer_size_for(core_type); + assert_eq!( + f.payload_len, + f.length - tensogram::wire::FRAME_HEADER_SIZE - footer, + "payload_len excludes the 16-byte header and the footer" + ); + let content_start = f.offset + tensogram::wire::FRAME_HEADER_SIZE; + assert_eq!(f.payload, unsafe { msg.as_ptr().add(content_start) }); + let payload = unsafe { slice::from_raw_parts(f.payload, f.payload_len) }; + assert_eq!(payload, &msg[content_start..f.offset + f.length - footer]); + } + } + + #[test] + fn ffi_frame_payload_stays_valid_after_the_iterator_is_freed() { + // Documented contract: `out->payload` points INTO the caller's `msg` + // and lives as long as `msg` does — freeing the cursor does not + // invalidate it. + let msg = frame_walk_buffered_message(1); + let fs = ffi_collect_frames(&msg); // frees the iterator before returning + let f = fs[0]; + assert!(f.payload_len > 0); + let payload = unsafe { slice::from_raw_parts(f.payload, f.payload_len) }; + let content_start = f.offset + tensogram::wire::FRAME_HEADER_SIZE; + assert_eq!(payload, &msg[content_start..content_start + f.payload_len]); + } + + #[test] + fn ffi_buffered_frame_set_matches_the_message_header_exactly() { + // Buffered mode: the encoder knows the whole message up front, so every + // preamble flag is an exact statement about the frames present. + let msg = frame_walk_buffered_message(2); + let mut h = blank_message_header(); + assert!(matches!( + super::tgm_message_header(msg.as_ptr(), msg.len(), &mut h), + super::TgmError::Ok + )); + let types: Vec = ffi_collect_frames(&msg) + .iter() + .map(|f| f.frame_type) + .collect(); + let has = |t: super::TgmFrameType| types.contains(&t); + assert_eq!( + h.has_header_metadata, + has(super::TgmFrameType::HeaderMetadata) + ); + assert_eq!( + h.has_footer_metadata, + has(super::TgmFrameType::FooterMetadata) + ); + assert_eq!(h.has_header_index, has(super::TgmFrameType::HeaderIndex)); + assert_eq!(h.has_footer_index, has(super::TgmFrameType::FooterIndex)); + assert_eq!(h.has_header_hashes, has(super::TgmFrameType::HeaderHash)); + assert_eq!(h.has_footer_hashes, has(super::TgmFrameType::FooterHash)); + assert_eq!( + h.has_preceder_metadata, + has(super::TgmFrameType::PrecederMetadata) + ); + } + + #[test] + fn ffi_streaming_message_header_never_understates_its_frames() { + // Streaming mode writes the preamble before any object, so it sets + // PRECEDER_METADATA advisorily even with no preceder frame. Only + // "frame present => flag set" holds in that direction. + let msg = frame_walk_streamed_message(); + let mut h = blank_message_header(); + assert!(matches!( + super::tgm_message_header(msg.as_ptr(), msg.len(), &mut h), + super::TgmError::Ok + )); + let types: Vec = ffi_collect_frames(&msg) + .iter() + .map(|f| f.frame_type) + .collect(); + let has = |t: super::TgmFrameType| types.contains(&t); + let implies = |present: bool, flag: bool| !present || flag; + assert!(implies( + has(super::TgmFrameType::HeaderMetadata), + h.has_header_metadata + )); + assert!(implies( + has(super::TgmFrameType::FooterMetadata), + h.has_footer_metadata + )); + assert!(implies( + has(super::TgmFrameType::HeaderIndex), + h.has_header_index + )); + assert!(implies( + has(super::TgmFrameType::FooterIndex), + h.has_footer_index + )); + assert!(implies( + has(super::TgmFrameType::HeaderHash), + h.has_header_hashes + )); + assert!(implies( + has(super::TgmFrameType::FooterHash), + h.has_footer_hashes + )); + assert!(implies( + has(super::TgmFrameType::PrecederMetadata), + h.has_preceder_metadata + )); + // The walk still stops cleanly at the postamble even though + // `total_length` was never back-filled. + assert!( + has(super::TgmFrameType::FooterIndex), + "footer frames are reachable" + ); + } + + #[test] + fn ffi_frame_iter_is_safe_past_exhaustion() { + let msg = frame_walk_buffered_message(1); + let it = super::tgm_frame_iter_create(msg.as_ptr(), msg.len()); + assert!(!it.is_null()); + let mut f = blank_frame(); + let mut n = 0usize; + while super::tgm_frame_iter_next(it, &mut f) { + n += 1; + } + assert!(n > 0, "walked at least one frame"); + // Calling next past the end is safe, stays false, and records no error. + assert!(!super::tgm_frame_iter_next(it, &mut f)); + assert!(!super::tgm_frame_iter_next(it, &mut f)); + assert!(super::tgm_last_error().is_null()); + super::tgm_frame_iter_free(it); + } + + #[test] + fn ffi_frame_iter_reports_a_malformed_frame_chain() { + let msg = frame_walk_buffered_message(2); + let last = *ffi_collect_frames(&msg).last().expect("at least one frame"); + // Cut *inside* the last frame: the preamble still parses, the chain does not. + let truncated = &msg[..last.offset + 8]; + let it = super::tgm_frame_iter_create(truncated.as_ptr(), truncated.len()); + assert!(!it.is_null(), "the preamble still parses"); + let mut f = blank_frame(); + let mut n = 0usize; + while super::tgm_frame_iter_next(it, &mut f) { + n += 1; + } + assert!(n > 0, "the frames before the cut are still yielded"); + // Stopping on a malformed frame is distinguishable from a clean end. + assert!( + !super::tgm_last_error().is_null(), + "a malformed frame must record why" + ); + // The cursor stays safe afterwards. + assert!(!super::tgm_frame_iter_next(it, &mut f)); + super::tgm_frame_iter_free(it); + } + + #[test] + fn ffi_frame_iter_rejects_a_buffer_that_is_not_a_message() { + let junk = b"not a tensogram message at all!!!!!!!!!!"; + let it = super::tgm_frame_iter_create(junk.as_ptr(), junk.len()); + assert!(it.is_null(), "a bad preamble yields no iterator"); + assert!(!super::tgm_last_error().is_null()); + + // A message truncated inside its preamble is rejected the same way. + let msg = frame_walk_buffered_message(1); + assert!(super::tgm_frame_iter_create(msg.as_ptr(), 8).is_null()); + } + + #[test] + fn ffi_frame_iter_null_safety() { + assert!(super::tgm_frame_iter_create(ptr::null(), 0).is_null()); + assert!(!super::tgm_last_error().is_null()); + + let mut f = blank_frame(); + assert!(!super::tgm_frame_iter_next(ptr::null_mut(), &mut f)); + assert!(f.payload.is_null(), "a rejected call writes nothing"); + + let msg = frame_walk_buffered_message(1); + let it = super::tgm_frame_iter_create(msg.as_ptr(), msg.len()); + assert!(!it.is_null()); + assert!(!super::tgm_frame_iter_next(it, ptr::null_mut())); + // The rejected call did not consume a frame. + assert!(super::tgm_frame_iter_next(it, &mut f)); + assert!(!f.payload.is_null()); + super::tgm_frame_iter_free(it); + + // free(NULL) is a no-op. + super::tgm_frame_iter_free(ptr::null_mut()); + } + + #[test] + fn ffi_frame_has_hash_mirrors_the_per_frame_flag() { + let msg = frame_walk_buffered_message(1); + let fs = ffi_collect_frames(&msg); + assert!( + fs.iter().any(|f| super::tgm_frame_has_hash(f)), + "default encode hashes every frame" + ); + for f in &fs { + // HASH_PRESENT is bit 1 of the raw frame flags. + assert_eq!(super::tgm_frame_has_hash(f), f.flags & (1 << 1) != 0); + } + } + + #[test] + fn ffi_frame_has_hash_null_safety() { + assert!(!super::tgm_frame_has_hash(ptr::null())); + } + + // ===================================================================== + // Synchronous remote access (tgm_is_remote_url / tgm_file_open_remote) + // + // Every test here runs in both builds. The ones whose *answer* depends + // on the `remote` Cargo feature say so with `cfg!(feature = "remote")` + // rather than being compiled out, so the feature-off stub path is + // covered by the default `cargo test -p tensogram-ffi --lib` run. + // ===================================================================== + + #[test] + fn ffi_is_remote_url_recognises_every_object_store_scheme() { + for url in [ + "s3://bucket/key.tgm", + "s3a://bucket/key.tgm", + "gs://bucket/key.tgm", + "az://container/blob.tgm", + "azure://container/blob.tgm", + "http://host/file.tgm", + "https://host/file.tgm", + // Scheme matching is case-insensitive in the core predicate. + "S3://BUCKET/KEY.TGM", + "HTTPS://HOST/FILE.TGM", + ] { + let c = CString::new(url).unwrap(); + assert_eq!( + super::tgm_is_remote_url(c.as_ptr()), + cfg!(feature = "remote"), + "{url}: a remote-capable build recognises it, a stub build cannot" + ); + } + } + + #[test] + fn ffi_is_remote_url_rejects_local_paths_in_every_build() { + for source in [ + "/tmp/local.tgm", + "relative/path.tgm", + "local.tgm", + // `file://` is a URL but not one of the remote schemes: the + // local backend already handles it. + "file:///tmp/local.tgm", + "s3:/one-slash-is-not-a-url", + "", + "not a url at all", + ] { + let c = CString::new(source).unwrap(); + assert!( + !super::tgm_is_remote_url(c.as_ptr()), + "{source} must not be treated as remote" + ); + } + } + + #[test] + fn ffi_is_remote_url_null_and_non_utf8_are_false_with_a_reason() { + assert!(!super::tgm_is_remote_url(ptr::null())); + let msg = unsafe { CStr::from_ptr(super::tgm_last_error()) } + .to_str() + .unwrap(); + assert!(!msg.is_empty(), "a NULL source is reported"); + + // Invalid UTF-8 cannot be a URL either — reported, never a panic. + let bad = [0xffu8, 0xfe, 0x00]; + assert!(!super::tgm_is_remote_url(bad.as_ptr() as *const c_char)); + assert!(!super::tgm_last_error().is_null()); + } + + #[cfg(not(feature = "remote"))] + #[test] + fn ffi_is_remote_url_without_the_feature_explains_itself() { + let c = CString::new("s3://bucket/key.tgm").unwrap(); + assert!(!super::tgm_is_remote_url(c.as_ptr())); + let msg = unsafe { CStr::from_ptr(super::tgm_last_error()) } + .to_str() + .unwrap(); + assert!(msg.contains("remote"), "{msg}"); + assert!(msg.contains("--features"), "names the fix: {msg}"); + } + + /// Two encoded messages on disk plus the `file://` URL that addresses + /// them. The `TempDir` is returned so the caller keeps it alive — it + /// deletes the directory on drop. `file://` goes through the same + /// object-store code path as `s3://` & friends, so it exercises the real + /// remote backend without a network. + #[cfg(feature = "remote")] + fn remote_fixture_file(values: &[f32]) -> (tempfile::TempDir, String) { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("remote_fixture.tgm"); + let mut bytes = ffi_encode_single_f32_tensor(values, ""); + bytes.extend_from_slice(&ffi_encode_single_f32_tensor(values, "")); + std::fs::write(&path, &bytes).expect("write fixture"); + let url = format!("file://{}", path.display()); + (dir, url) + } + + #[test] + fn ffi_file_open_remote_rejects_null_source_and_out() { + // Argument validation runs before the feature check, so a C caller + // (and the C++ / Fortran wrappers) get the same answer from every + // build of the library. + let mut file: *mut super::TgmFile = ptr::null_mut(); + let err = super::tgm_file_open_remote( + ptr::null(), + ptr::null(), + ptr::null(), + 0, + ptr::null(), + &mut file, + ); + assert!(matches!(err, super::TgmError::InvalidArg)); + assert!(file.is_null(), "the out-param is left alone"); + + let src = CString::new("s3://bucket/key.tgm").unwrap(); + let err = super::tgm_file_open_remote( + src.as_ptr(), + ptr::null(), + ptr::null(), + 0, + ptr::null(), + ptr::null_mut(), + ); + assert!(matches!(err, super::TgmError::InvalidArg)); + } + + #[test] + fn ffi_file_open_remote_rejects_mismatched_option_arrays() { + let src = CString::new("s3://bucket/key.tgm").unwrap(); + let key = CString::new("aws_region").unwrap(); + let value = CString::new("eu-west-1").unwrap(); + let keys = [key.as_ptr()]; + let values = [value.as_ptr()]; + let mut file: *mut super::TgmFile = ptr::null_mut(); + + // n_options > 0 but the key array is missing. + let err = super::tgm_file_open_remote( + src.as_ptr(), + ptr::null(), + values.as_ptr(), + 1, + ptr::null(), + &mut file, + ); + assert!(matches!(err, super::TgmError::InvalidArg)); + + // …or the value array is missing. + let err = super::tgm_file_open_remote( + src.as_ptr(), + keys.as_ptr(), + ptr::null(), + 1, + ptr::null(), + &mut file, + ); + assert!(matches!(err, super::TgmError::InvalidArg)); + + // …or an individual entry is NULL. + let null_entry: [*const c_char; 1] = [ptr::null()]; + let err = super::tgm_file_open_remote( + src.as_ptr(), + null_entry.as_ptr(), + values.as_ptr(), + 1, + ptr::null(), + &mut file, + ); + assert!(matches!(err, super::TgmError::InvalidArg)); + let err = super::tgm_file_open_remote( + src.as_ptr(), + keys.as_ptr(), + null_entry.as_ptr(), + 1, + ptr::null(), + &mut file, + ); + assert!(matches!(err, super::TgmError::InvalidArg)); + + // …or `n_options` overruns the arrays the caller actually built: + // entry 1 of a 1-element array is out of bounds, which the caller + // must not do — but a *shorter* declared count is always safe. + let err = super::tgm_file_open_remote( + src.as_ptr(), + keys.as_ptr(), + values.as_ptr(), + 0, + ptr::null(), + &mut file, + ); + assert!( + !matches!(err, super::TgmError::InvalidArg), + "n_options = 0 ignores the arrays entirely" + ); + assert!(file.is_null(), "no handle is produced by a failed open"); + } + + #[cfg(not(feature = "remote"))] + #[test] + fn ffi_file_open_remote_without_the_feature_is_a_clear_remote_error() { + let src = CString::new("s3://bucket/key.tgm").unwrap(); + let mut file: *mut super::TgmFile = ptr::null_mut(); + let err = super::tgm_file_open_remote( + src.as_ptr(), + ptr::null(), + ptr::null(), + 0, + ptr::null(), + &mut file, + ); + assert!(matches!(err, super::TgmError::Remote)); + assert!(file.is_null()); + let msg = unsafe { CStr::from_ptr(super::tgm_last_error()) } + .to_str() + .unwrap(); + assert!(msg.contains("tgm_file_open_remote"), "{msg}"); + assert!(msg.contains("remote"), "{msg}"); + assert!(msg.contains("--features"), "names the fix: {msg}"); + } + + #[cfg(feature = "remote")] + #[test] + fn ffi_file_open_remote_round_trips_through_the_ordinary_file_api() { + let values = [1.5f32, 2.5, 3.5]; + let (_dir, url) = remote_fixture_file(&values); + let c_url = CString::new(url.clone()).unwrap(); + + // NULL options => library defaults. + let mut file: *mut super::TgmFile = ptr::null_mut(); + let err = super::tgm_file_open_remote( + c_url.as_ptr(), + ptr::null(), + ptr::null(), + 0, + ptr::null(), + &mut file, + ); + assert!(matches!(err, super::TgmError::Ok), "open {url} failed"); + assert!(!file.is_null()); + + // The handle is a plain `tgm_file_t`: the whole sync reader API works. + let mut count: usize = 0; + assert!(matches!( + super::tgm_file_message_count(file, &mut count), + super::TgmError::Ok + )); + assert_eq!(count, 2); + + let mut raw = zeroed_bytes(); + assert!(matches!( + super::tgm_file_read_message(file, 1, &mut raw), + super::TgmError::Ok + )); + assert!(take_bytes(raw).len() > tensogram::wire::PREAMBLE_SIZE); + + let mut msg: *mut super::TgmMessage = ptr::null_mut(); + assert!(matches!( + super::tgm_file_decode_message(file, 0, 0, 0, 0, &mut msg), + super::TgmError::Ok + )); + assert_eq!(super::tgm_message_num_objects(msg), 1); + let mut len = 0usize; + let data = super::tgm_object_data(msg, 0, &mut len); + let decoded: Vec = unsafe { slice::from_raw_parts(data, len) } + .chunks_exact(4) + .map(|c| f32::from_ne_bytes(c.try_into().unwrap())) + .collect(); + assert_eq!(decoded, values); + super::tgm_message_free(msg); + + assert_eq!( + unsafe { CStr::from_ptr(super::tgm_file_path(file)) } + .to_str() + .unwrap(), + url, + "the handle remembers the source it was opened from" + ); + super::tgm_file_close(file); + } + + #[cfg(feature = "remote")] + #[test] + fn ffi_file_open_remote_accepts_scan_options_and_storage_options() { + let values = [7.0f32, 8.0]; + let (_dir, url) = remote_fixture_file(&values); + let c_url = CString::new(url).unwrap(); + let key = CString::new("some_storage_key").unwrap(); + let value = CString::new("some_value").unwrap(); + let keys = [key.as_ptr()]; + let vals = [value.as_ptr()]; + + for bidirectional in [true, false] { + let opts = super::TgmRemoteScanOptions { bidirectional }; + let mut file: *mut super::TgmFile = ptr::null_mut(); + let err = super::tgm_file_open_remote( + c_url.as_ptr(), + keys.as_ptr(), + vals.as_ptr(), + 1, + &opts, + &mut file, + ); + assert!( + matches!(err, super::TgmError::Ok), + "bidirectional = {bidirectional}" + ); + let mut count: usize = 0; + assert!(matches!( + super::tgm_file_message_count(file, &mut count), + super::TgmError::Ok + )); + assert_eq!(count, 2, "both walkers discover the same messages"); + super::tgm_file_close(file); + } + } + + #[cfg(feature = "remote")] + #[test] + fn ffi_file_open_remote_maps_failures_to_remote_error() { + let mut file: *mut super::TgmFile = ptr::null_mut(); + + // A URL that parses but addresses nothing. + let missing = CString::new("file:///nonexistent/definitely/not/here.tgm").unwrap(); + let err = super::tgm_file_open_remote( + missing.as_ptr(), + ptr::null(), + ptr::null(), + 0, + ptr::null(), + &mut file, + ); + assert!(matches!(err, super::TgmError::Remote), "{err:?}"); + assert!(file.is_null(), "no handle on failure"); + assert!(!super::tgm_last_error().is_null()); + + // A source that is not a URL at all. + let not_a_url = CString::new("/tmp/plain/path.tgm").unwrap(); + let err = super::tgm_file_open_remote( + not_a_url.as_ptr(), + ptr::null(), + ptr::null(), + 0, + ptr::null(), + &mut file, + ); + assert!(matches!(err, super::TgmError::Remote), "{err:?}"); + + // The storage options really do reach the backend: a known key with + // an unparseable value is rejected before any network I/O happens. + let s3 = CString::new("s3://bucket/key.tgm").unwrap(); + let key = CString::new("aws_virtual_hosted_style_request").unwrap(); + let value = CString::new("not-a-bool").unwrap(); + let keys = [key.as_ptr()]; + let vals = [value.as_ptr()]; + let err = super::tgm_file_open_remote( + s3.as_ptr(), + keys.as_ptr(), + vals.as_ptr(), + 1, + ptr::null(), + &mut file, + ); + assert!(matches!(err, super::TgmError::Remote), "{err:?}"); + let msg = unsafe { CStr::from_ptr(super::tgm_last_error()) } + .to_str() + .unwrap(); + assert!( + msg.contains("not-a-bool"), + "the option was forwarded: {msg}" + ); + } + + // ===================================================================== + // Typed enums: tgm_dtype, tgm_byte_order, tgm_aggregate_hash_policy, + // tgm_compression_backend. + // + // Every mirror test maps its core enum through an *exhaustive* `match` + // with no wildcard arm, so adding a variant to the core type stops this + // crate compiling until the C enum, its conversion and the mirror all + // learn about it. That is a stronger drift guard than grepping the + // generated header: it fails at build time, on the commit that + // introduces the variant. + // ===================================================================== + + #[test] + fn ffi_dtype_enum_mirrors_every_core_dtype_variant() { + use tensogram::Dtype; + const ALL: [Dtype; 15] = [ + Dtype::Float16, + Dtype::Bfloat16, + Dtype::Float32, + Dtype::Float64, + Dtype::Complex64, + Dtype::Complex128, + Dtype::Int8, + Dtype::Int16, + Dtype::Int32, + Dtype::Int64, + Dtype::Uint8, + Dtype::Uint16, + Dtype::Uint32, + Dtype::Uint64, + Dtype::Bitmask, + ]; + for core in ALL { + let expected = match core { + Dtype::Float16 => super::TgmDtype::Float16, + Dtype::Bfloat16 => super::TgmDtype::Bfloat16, + Dtype::Float32 => super::TgmDtype::Float32, + Dtype::Float64 => super::TgmDtype::Float64, + Dtype::Complex64 => super::TgmDtype::Complex64, + Dtype::Complex128 => super::TgmDtype::Complex128, + Dtype::Int8 => super::TgmDtype::Int8, + Dtype::Int16 => super::TgmDtype::Int16, + Dtype::Int32 => super::TgmDtype::Int32, + Dtype::Int64 => super::TgmDtype::Int64, + Dtype::Uint8 => super::TgmDtype::Uint8, + Dtype::Uint16 => super::TgmDtype::Uint16, + Dtype::Uint32 => super::TgmDtype::Uint32, + Dtype::Uint64 => super::TgmDtype::Uint64, + Dtype::Bitmask => super::TgmDtype::Bitmask, + }; + assert_eq!(super::TgmDtype::from(core), expected, "{core}"); + } + + // The mapping is injective: 15 core variants, 15 distinct C codes, + // densely numbered from 0 in core declaration order. + let codes: Vec = ALL + .iter() + .map(|d| super::TgmDtype::from(*d) as i32) + .collect(); + assert_eq!(codes, (0..15).collect::>()); + // Frozen ABI values — the C++ / Fortran mirrors hard-code them. + assert_eq!(super::TgmDtype::Float16 as i32, 0); + assert_eq!(super::TgmDtype::Bitmask as i32, 14); + } + + #[test] + fn ffi_byte_order_enum_mirrors_every_core_variant() { + use tensogram::ByteOrder; + for core in [ByteOrder::Little, ByteOrder::Big] { + let expected = match core { + ByteOrder::Little => super::TgmByteOrder::Little, + ByteOrder::Big => super::TgmByteOrder::Big, + }; + assert_eq!(super::TgmByteOrder::from(core), expected, "{core:?}"); + } + assert_eq!(super::TgmByteOrder::Little as i32, 0); + assert_eq!(super::TgmByteOrder::Big as i32, 1); + } + + #[test] + fn ffi_aggregate_hash_policy_enum_mirrors_every_core_variant() { + use tensogram::AggregateHashPolicy as Core; + for core in [ + Core::Auto, + Core::None, + Core::Header, + Core::Footer, + Core::Both, + ] { + let c = match core { + Core::Auto => super::TgmAggregateHashPolicy::Auto, + Core::None => super::TgmAggregateHashPolicy::None, + Core::Header => super::TgmAggregateHashPolicy::Header, + Core::Footer => super::TgmAggregateHashPolicy::Footer, + Core::Both => super::TgmAggregateHashPolicy::Both, + }; + assert_eq!(Core::from(c), core, "{core:?} must round-trip"); + } + assert_eq!(super::TgmAggregateHashPolicy::Auto as i32, 0); + assert_eq!(super::TgmAggregateHashPolicy::Both as i32, 4); + // `Auto` is the zero value, so a zero-initialised C struct means + // "let the encoder decide" — the same default the Rust API has. + assert_eq!( + Core::from(super::TgmAggregateHashPolicy::Auto), + Core::default() + ); + } + + #[test] + fn ffi_compression_backend_enum_mirrors_every_core_variant() { + use tensogram::CompressionBackend as Core; + for core in [Core::Auto, Core::Ffi, Core::Pure] { + let c = match core { + Core::Auto => super::TgmCompressionBackend::Auto, + Core::Ffi => super::TgmCompressionBackend::Ffi, + Core::Pure => super::TgmCompressionBackend::Pure, + }; + assert_eq!(Core::from(c), core, "{core:?} must round-trip"); + } + assert_eq!(super::TgmCompressionBackend::Auto as i32, 0); + assert_eq!(super::TgmCompressionBackend::Pure as i32, 2); + assert_eq!( + Core::from(super::TgmCompressionBackend::Auto), + Core::default(), + "zero-initialised C structs get the library default" + ); + } + + // ── typed accessors: tgm_object_dtype_enum / tgm_object_byte_order_enum ── + + /// Encode one 8-element object per supplied descriptor spec and decode + /// the message back through the C API, so the typed accessors are + /// exercised against a real decoded message. + fn decode_objects_via_ffi( + specs: &[(tensogram::Dtype, tensogram::ByteOrder)], + ) -> *mut super::TgmMessage { + const N: usize = 8; + let descs: Vec = specs + .iter() + .map(|(dtype, byte_order)| DataObjectDescriptor { + obj_type: "ntensor".to_string(), + ndim: 1, + shape: vec![N as u64], + strides: vec![1], + dtype: *dtype, + byte_order: *byte_order, + encoding: "none".to_string(), + filter: "none".to_string(), + compression: "none".to_string(), + params: BTreeMap::new(), + masks: None, + }) + .collect(); + // A bitmask packs one bit per element; everything else is dense. + let datas: Vec> = specs + .iter() + .map(|(dtype, _)| match dtype { + tensogram::Dtype::Bitmask => vec![0u8; N.div_ceil(8)], + other => vec![0u8; N * other.byte_width()], + }) + .collect(); + let pairs: Vec<(&DataObjectDescriptor, &[u8])> = descs + .iter() + .zip(datas.iter()) + .map(|(d, v)| (d, v.as_slice())) + .collect(); + let msg = encode( + &GlobalMetadata::default(), + &pairs, + &EncodeOptions::default(), + ) + .expect("encode fixture"); + + let mut out: *mut super::TgmMessage = ptr::null_mut(); + let err = super::tgm_decode(msg.as_ptr(), msg.len(), 0, 0, 0, &mut out); + assert!(matches!(err, super::TgmError::Ok), "decode fixture"); + out + } + + /// The string a getter returned, or `None` for the NULL that marks an + /// invalid handle / index. + fn cstr_opt(p: *const c_char) -> Option { + if p.is_null() { + None + } else { + Some( + unsafe { CStr::from_ptr(p) } + .to_str() + .expect("valid UTF-8") + .to_string(), + ) + } + } + + #[test] + fn ffi_object_dtype_enum_agrees_with_the_string_getter_for_every_dtype() { + use tensogram::Dtype; + let all = [ + Dtype::Float16, + Dtype::Bfloat16, + Dtype::Float32, + Dtype::Float64, + Dtype::Complex64, + Dtype::Complex128, + Dtype::Int8, + Dtype::Int16, + Dtype::Int32, + Dtype::Int64, + Dtype::Uint8, + Dtype::Uint16, + Dtype::Uint32, + Dtype::Uint64, + Dtype::Bitmask, + ]; + let specs: Vec<_> = all + .iter() + .map(|d| (*d, tensogram::ByteOrder::native())) + .collect(); + let msg = decode_objects_via_ffi(&specs); + assert_eq!(super::tgm_message_num_objects(msg), all.len()); + + for (i, dtype) in all.iter().enumerate() { + let via_enum = super::tgm_object_dtype_enum(msg, i); + let via_string = cstr_opt(super::tgm_object_dtype(msg, i)).expect("string getter"); + assert_eq!( + via_enum, + super::TgmDtype::from(*dtype), + "object {i}: {dtype} decoded to the wrong code" + ); + // The two getters describe the same object: the string getter + // is `Dtype::to_string()`, so it must name the enum's variant. + assert_eq!( + via_string, + dtype.to_string(), + "object {i}: string getter drifted" + ); + } + super::tgm_message_free(msg); + } + + #[test] + fn ffi_object_byte_order_enum_agrees_with_the_string_getter() { + use tensogram::{ByteOrder, Dtype}; + let specs = [ + (Dtype::Uint8, ByteOrder::Little), + (Dtype::Uint8, ByteOrder::Big), + ]; + let msg = decode_objects_via_ffi(&specs); + for (i, (_, order)) in specs.iter().enumerate() { + let via_enum = super::tgm_object_byte_order_enum(msg, i); + let via_string = cstr_opt(super::tgm_object_byte_order(msg, i)).expect("string getter"); + assert_eq!(via_enum, super::TgmByteOrder::from(*order), "object {i}"); + let expected_string = match order { + ByteOrder::Little => "little", + ByteOrder::Big => "big", + }; + assert_eq!(via_string, expected_string, "object {i}"); + } + super::tgm_message_free(msg); + } + + #[test] + fn ffi_object_enum_accessors_are_null_and_out_of_bounds_safe() { + let specs = [(tensogram::Dtype::Float32, tensogram::ByteOrder::native())]; + let msg = decode_objects_via_ffi(&specs); + let n = super::tgm_message_num_objects(msg); + assert_eq!(n, 1); + + let msg_const: *const super::TgmMessage = msg; + for (handle, index) in [ + (msg_const, n), // first out-of-range index + (msg_const, usize::MAX), // wildly out of range + (ptr::null(), 0), // NULL handle + (ptr::null(), 4_096), // both wrong + ] { + // The documented fallback is the zero-valued variant … + assert_eq!( + super::tgm_object_dtype_enum(handle, index), + super::TgmDtype::Float16 + ); + assert_eq!( + super::tgm_object_byte_order_enum(handle, index), + super::TgmByteOrder::Little + ); + // … and the reason is always reported, while the paired string + // getter returns NULL — the unambiguous way to bounds-check. + assert!(!super::tgm_last_error().is_null()); + assert!(cstr_opt(super::tgm_object_dtype(handle, index)).is_none()); + assert!(cstr_opt(super::tgm_object_byte_order(handle, index)).is_none()); + } + super::tgm_message_free(msg); + } + + // ===================================================================== + // TgmEncodeOptions and the entry points that take it. + // + // Encoded messages embed a fresh UUID and timestamp in their reserved + // provenance metadata, so "same options ⇒ same output" is asserted + // structurally — identical frame sequences and identical decoded + // payloads — never by byte equality. + // ===================================================================== + + /// A `TgmEncodeOptions` holding nothing but defaults: no hash, `AUTO` + /// placement, `AUTO` backend, non-finite values rejected, every method + /// string NULL and the negative "use the library default" threshold. + fn blank_encode_options() -> super::TgmEncodeOptions { + super::TgmEncodeOptions { + hash: ptr::null(), + aggregate_hash: super::TgmAggregateHashPolicy::Auto, + compression_backend: super::TgmCompressionBackend::Auto, + allow_nan: false, + allow_inf: false, + nan_mask_method: ptr::null(), + pos_inf_mask_method: ptr::null(), + neg_inf_mask_method: ptr::null(), + small_mask_threshold_bytes: -1, + } + } + + /// One-object encode JSON with a caller-chosen compression stage. + fn encode_json_with_compression(n: usize, compression: &str) -> CString { + CString::new(format!( + r#"{{"descriptors":[{{"type":"ntensor","ndim":1,"shape":[{n}],"strides":[1],"dtype":"float32","byte_order":"{bo}","encoding":"none","filter":"none","compression":"{compression}"}}]}}"#, + bo = if cfg!(target_endian = "little") { + "little" + } else { + "big" + }, + )) + .unwrap() + } + + /// Encode `values` through `tgm_encode_with_encode_options`, returning + /// the raw error code and (on success) the message bytes. + fn encode_with_encode_options( + values: &[f32], + compression: &str, + opts: *const super::TgmEncodeOptions, + ) -> (super::TgmError, Vec) { + let json = encode_json_with_compression(values.len(), compression); + let data: Vec = values.iter().flat_map(|v| v.to_ne_bytes()).collect(); + let data_ptr: *const u8 = data.as_ptr(); + let data_len: usize = data.len(); + let mut out = zeroed_bytes(); + let err = super::tgm_encode_with_encode_options( + json.as_ptr(), + &data_ptr as *const *const u8, + &data_len as *const usize, + 1, + 0, + opts, + &mut out, + ); + if matches!(err, super::TgmError::Ok) { + (err, take_bytes(out)) + } else { + (err, Vec::new()) + } + } + + /// The frame types of `msg`, in order, via the C frame walker. + fn ffi_frame_type_sequence(msg: &[u8]) -> Vec { + ffi_collect_frames(msg) + .iter() + .map(|f| f.frame_type) + .collect() + } + + /// The single object's decoded f32 payload, via the C decode API. + fn ffi_decode_f32s(msg: &[u8]) -> Vec { + let mut handle: *mut super::TgmMessage = ptr::null_mut(); + let err = super::tgm_decode(msg.as_ptr(), msg.len(), 1, 0, 0, &mut handle); + assert!(matches!(err, super::TgmError::Ok), "decode failed: {err:?}"); + let mut len = 0usize; + let data = super::tgm_object_data(handle, 0, &mut len); + let out: Vec = unsafe { slice::from_raw_parts(data, len) } + .chunks_exact(4) + .map(|c| f32::from_ne_bytes(c.try_into().unwrap())) + .collect(); + super::tgm_message_free(handle); + out + } + + #[test] + fn ffi_encode_with_encode_options_null_matches_the_existing_default_encode() { + let values = [1.0f32, 2.0, 3.0, 4.0]; + let baseline = ffi_encode_single_f32_tensor(&values, ""); + let (err, with_opts) = encode_with_encode_options(&values, "none", ptr::null()); + assert!(matches!(err, super::TgmError::Ok)); + + assert_eq!( + ffi_frame_type_sequence(&with_opts), + ffi_frame_type_sequence(&baseline), + "NULL options must reproduce tgm_encode's default frame layout" + ); + assert_eq!(ffi_decode_f32s(&with_opts), values.to_vec()); + + // An all-defaults struct is the same thing as NULL. + let blank = blank_encode_options(); + let (err, explicit) = encode_with_encode_options(&values, "none", &blank); + assert!(matches!(err, super::TgmError::Ok)); + assert_eq!( + ffi_frame_type_sequence(&explicit), + ffi_frame_type_sequence(&baseline) + ); + } + + #[test] + fn ffi_encode_with_encode_options_hash_field_turns_hashing_on() { + let values = [1.0f32, 2.0]; + let mut opts = blank_encode_options(); + let (err, unhashed) = encode_with_encode_options(&values, "none", &opts); + assert!(matches!(err, super::TgmError::Ok)); + + let algo = CString::new("xxh3").unwrap(); + opts.hash = algo.as_ptr(); + let (err, hashed) = encode_with_encode_options(&values, "none", &opts); + assert!(matches!(err, super::TgmError::Ok)); + + let header_of = |msg: &[u8]| { + let mut h = blank_message_header(); + assert!(matches!( + super::tgm_message_header(msg.as_ptr(), msg.len(), &mut h), + super::TgmError::Ok + )); + h + }; + assert!(!header_of(&unhashed).has_hashes_present, "NULL hash is off"); + assert!( + header_of(&hashed).has_hashes_present, + "\"xxh3\" turns it on" + ); + assert_eq!(ffi_decode_f32s(&hashed), values.to_vec()); + } + + #[test] + fn ffi_encode_with_encode_options_aggregate_policy_places_the_hash_frames() { + use super::TgmAggregateHashPolicy as Policy; + use super::TgmFrameType::{FooterHash, HeaderHash}; + let values = [1.0f32, 2.0, 3.0]; + let algo = CString::new("xxh3").unwrap(); + + // (policy, expects a header hash frame, expects a footer hash frame) + for (policy, want_header, want_footer) in [ + (Policy::Auto, true, false), + (Policy::None, false, false), + (Policy::Header, true, false), + (Policy::Footer, false, true), + (Policy::Both, true, true), + ] { + let mut opts = blank_encode_options(); + opts.hash = algo.as_ptr(); + opts.aggregate_hash = policy; + let (err, msg) = encode_with_encode_options(&values, "none", &opts); + assert!(matches!(err, super::TgmError::Ok), "{policy:?}"); + + // Verified with the frame walker: the aggregate hash frames are + // exactly where the policy says they should be. + let frames = ffi_frame_type_sequence(&msg); + assert_eq!( + frames.contains(&HeaderHash), + want_header, + "{policy:?}: header hash frame, got {frames:?}" + ); + assert_eq!( + frames.contains(&FooterHash), + want_footer, + "{policy:?}: footer hash frame, got {frames:?}" + ); + assert_eq!(ffi_decode_f32s(&msg), values.to_vec(), "{policy:?}"); + } + } + + #[test] + fn ffi_encode_with_encode_options_carries_the_mask_fields() { + let values = [1.0f32, f32::NAN, 3.0]; + + // The default reject policy is unchanged. + let (err, _) = encode_with_encode_options(&values, "none", ptr::null()); + assert!( + matches!(err, super::TgmError::Encoding), + "NaN is rejected by default, got {err:?}" + ); + + // allow_nan + an explicit mask method get through to the encoder. + let method = CString::new("rle").unwrap(); + let mut opts = blank_encode_options(); + opts.allow_nan = true; + opts.nan_mask_method = method.as_ptr(); + opts.small_mask_threshold_bytes = 0; // disable the small-mask fallback + let (err, msg) = encode_with_encode_options(&values, "none", &opts); + assert!(matches!(err, super::TgmError::Ok), "{err:?}"); + let decoded = ffi_decode_f32s(&msg); + assert!(decoded[1].is_nan(), "the NaN came back: {decoded:?}"); + assert_eq!([decoded[0], decoded[2]], [1.0, 3.0]); + } + + #[test] + fn ffi_encode_with_encode_options_selects_the_compression_backend() { + use super::TgmCompressionBackend as Backend; + let values = [1.0f32, 2.0, 3.0, 4.0]; + for backend in [Backend::Auto, Backend::Ffi, Backend::Pure] { + let mut opts = blank_encode_options(); + opts.compression_backend = backend; + let (err, msg) = encode_with_encode_options(&values, "zstd", &opts); + assert!(matches!(err, super::TgmError::Ok), "{backend:?}: {err:?}"); + assert_eq!( + ffi_decode_f32s(&msg), + values.to_vec(), + "{backend:?} must produce a decodable message" + ); + } + } + + #[test] + fn ffi_encode_with_encode_options_rejects_bad_names_and_null_args() { + let values = [1.0f32]; + + let bad_hash = CString::new("md5").unwrap(); + let mut opts = blank_encode_options(); + opts.hash = bad_hash.as_ptr(); + let (err, _) = encode_with_encode_options(&values, "none", &opts); + assert!(matches!(err, super::TgmError::InvalidArg), "{err:?}"); + + let bad_method = CString::new("definitely-not-a-method").unwrap(); + let mut opts = blank_encode_options(); + opts.nan_mask_method = bad_method.as_ptr(); + let (err, _) = encode_with_encode_options(&values, "none", &opts); + assert!(matches!(err, super::TgmError::InvalidArg), "{err:?}"); + assert!(!super::tgm_last_error().is_null()); + + // NULL metadata / out are rejected exactly like tgm_encode's. + let mut out = zeroed_bytes(); + let err = super::tgm_encode_with_encode_options( + ptr::null(), + ptr::null(), + ptr::null(), + 0, + 0, + ptr::null(), + &mut out, + ); + assert!(matches!(err, super::TgmError::InvalidArg)); + let json = encode_json_with_compression(1, "none"); + let err = super::tgm_encode_with_encode_options( + json.as_ptr(), + ptr::null(), + ptr::null(), + 0, + 0, + ptr::null(), + ptr::null_mut(), + ); + assert!(matches!(err, super::TgmError::InvalidArg)); + } + + /// Read message `index` out of an open file and return its raw bytes. + fn ffi_read_message(file: *mut super::TgmFile, index: usize) -> Vec { + let mut raw = zeroed_bytes(); + let err = super::tgm_file_read_message(file, index, &mut raw); + assert!(matches!(err, super::TgmError::Ok), "read {index}: {err:?}"); + take_bytes(raw) + } + + #[test] + fn ffi_file_append_with_encode_options_round_trips_and_places_frames() { + use super::TgmFrameType::{FooterHash, HeaderHash}; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("append_encode_options.tgm"); + let c_path = CString::new(path.to_str().unwrap()).unwrap(); + + let mut file: *mut super::TgmFile = ptr::null_mut(); + assert!(matches!( + super::tgm_file_create(c_path.as_ptr(), &mut file), + super::TgmError::Ok + )); + + let values = [5.0f32, 6.0, 7.0]; + let json = encode_json_with_compression(values.len(), "none"); + let data: Vec = values.iter().flat_map(|v| v.to_ne_bytes()).collect(); + let data_ptr: *const u8 = data.as_ptr(); + let data_len: usize = data.len(); + + // Message 0: NULL options => the current defaults (no hashing). + let err = super::tgm_file_append_with_encode_options( + file, + json.as_ptr(), + &data_ptr as *const *const u8, + &data_len as *const usize, + 1, + 0, + ptr::null(), + ); + assert!(matches!(err, super::TgmError::Ok), "{err:?}"); + + // Message 1: hashed, with the aggregate frame written at both ends. + let algo = CString::new("xxh3").unwrap(); + let mut opts = blank_encode_options(); + opts.hash = algo.as_ptr(); + opts.aggregate_hash = super::TgmAggregateHashPolicy::Both; + let err = super::tgm_file_append_with_encode_options( + file, + json.as_ptr(), + &data_ptr as *const *const u8, + &data_len as *const usize, + 1, + 0, + &opts, + ); + assert!(matches!(err, super::TgmError::Ok), "{err:?}"); + super::tgm_file_close(file); + + let mut file: *mut super::TgmFile = ptr::null_mut(); + assert!(matches!( + super::tgm_file_open(c_path.as_ptr(), &mut file), + super::TgmError::Ok + )); + let mut count = 0usize; + assert!(matches!( + super::tgm_file_message_count(file, &mut count), + super::TgmError::Ok + )); + assert_eq!(count, 2); + + let defaults = ffi_frame_type_sequence(&ffi_read_message(file, 0)); + assert!(!defaults.contains(&HeaderHash), "{defaults:?}"); + assert!(!defaults.contains(&FooterHash), "{defaults:?}"); + + let both = ffi_frame_type_sequence(&ffi_read_message(file, 1)); + assert!(both.contains(&HeaderHash), "{both:?}"); + assert!(both.contains(&FooterHash), "{both:?}"); + + for index in 0..2 { + assert_eq!( + ffi_decode_f32s(&ffi_read_message(file, index)), + values.to_vec(), + "message {index}" + ); + } + super::tgm_file_close(file); + } + + #[test] + fn ffi_file_append_with_encode_options_rejects_null_args() { + let err = super::tgm_file_append_with_encode_options( + ptr::null_mut(), + ptr::null(), + ptr::null(), + ptr::null(), + 0, + 0, + ptr::null(), + ); + assert!(matches!(err, super::TgmError::InvalidArg)); + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("append_null_json.tgm"); + let c_path = CString::new(path.to_str().unwrap()).unwrap(); + let mut file: *mut super::TgmFile = ptr::null_mut(); + assert!(matches!( + super::tgm_file_create(c_path.as_ptr(), &mut file), + super::TgmError::Ok + )); + let err = super::tgm_file_append_with_encode_options( + file, + ptr::null(), + ptr::null(), + ptr::null(), + 0, + 0, + ptr::null(), + ); + assert!(matches!(err, super::TgmError::InvalidArg)); + super::tgm_file_close(file); + } + + /// Stream `values` as one object into `path` with the given options. + fn stream_one_object( + path: &std::path::Path, + values: &[f32], + opts: *const super::TgmEncodeOptions, + ) -> super::TgmError { + let c_path = CString::new(path.to_str().unwrap()).unwrap(); + let meta = CString::new(r#"{"descriptors":[]}"#).unwrap(); + let mut enc: *mut super::TgmStreamingEncoder = ptr::null_mut(); + let err = super::tgm_streaming_encoder_create_with_encode_options( + c_path.as_ptr(), + meta.as_ptr(), + 0, + opts, + &mut enc, + ); + if !matches!(err, super::TgmError::Ok) { + assert!(enc.is_null(), "no handle on failure"); + return err; + } + let desc = CString::new(format!( + r#"{{"type":"ntensor","ndim":1,"shape":[{n}],"strides":[1],"dtype":"float32","byte_order":"{bo}","encoding":"none","filter":"none","compression":"none"}}"#, + n = values.len(), + bo = if cfg!(target_endian = "little") { "little" } else { "big" }, + )) + .unwrap(); + let data: Vec = values.iter().flat_map(|v| v.to_ne_bytes()).collect(); + assert!(matches!( + super::tgm_streaming_encoder_write(enc, desc.as_ptr(), data.as_ptr(), data.len()), + super::TgmError::Ok + )); + assert!(matches!( + super::tgm_streaming_encoder_finish(enc), + super::TgmError::Ok + )); + super::tgm_streaming_encoder_free(enc); + super::TgmError::Ok + } + + #[test] + fn ffi_streaming_encoder_create_with_encode_options_round_trips() { + use super::TgmFrameType::{FooterHash, HeaderHash}; + let dir = tempfile::tempdir().unwrap(); + let values = [11.0f32, 22.0]; + + // NULL options => today's defaults. + let defaults_path = dir.path().join("stream_defaults.tgm"); + assert!(matches!( + stream_one_object(&defaults_path, &values, ptr::null()), + super::TgmError::Ok + )); + let raw = std::fs::read(&defaults_path).unwrap(); + let frames = ffi_frame_type_sequence(&raw); + assert!(!frames.contains(&HeaderHash), "{frames:?}"); + assert!(!frames.contains(&FooterHash), "{frames:?}"); + assert_eq!(ffi_decode_f32s(&raw), values.to_vec()); + + // Explicit footer placement is what streaming supports. + let algo = CString::new("xxh3").unwrap(); + let mut opts = blank_encode_options(); + opts.hash = algo.as_ptr(); + opts.aggregate_hash = super::TgmAggregateHashPolicy::Footer; + let footer_path = dir.path().join("stream_footer.tgm"); + assert!(matches!( + stream_one_object(&footer_path, &values, &opts), + super::TgmError::Ok + )); + let raw = std::fs::read(&footer_path).unwrap(); + let frames = ffi_frame_type_sequence(&raw); + assert!(frames.contains(&FooterHash), "{frames:?}"); + assert!( + !frames.contains(&HeaderHash), + "streaming never writes a header hash: {frames:?}" + ); + assert_eq!(ffi_decode_f32s(&raw), values.to_vec()); + } + + #[test] + fn ffi_streaming_encoder_create_with_encode_options_rejects_header_placements() { + let dir = tempfile::tempdir().unwrap(); + let algo = CString::new("xxh3").unwrap(); + for policy in [ + super::TgmAggregateHashPolicy::Header, + super::TgmAggregateHashPolicy::Both, + ] { + let mut opts = blank_encode_options(); + opts.hash = algo.as_ptr(); + opts.aggregate_hash = policy; + let path = dir.path().join(format!("stream_reject_{policy:?}.tgm")); + let err = stream_one_object(&path, &[1.0f32], &opts); + assert!( + matches!(err, super::TgmError::Encoding), + "{policy:?} must be rejected, got {err:?}" + ); + let msg = unsafe { CStr::from_ptr(super::tgm_last_error()) } + .to_str() + .unwrap(); + assert!(msg.contains("streaming"), "{policy:?}: {msg}"); + } + } + + #[test] + fn ffi_streaming_encoder_create_with_encode_options_rejects_null_args() { + let mut enc: *mut super::TgmStreamingEncoder = ptr::null_mut(); + let meta = CString::new(r#"{"descriptors":[]}"#).unwrap(); + let dir = tempfile::tempdir().unwrap(); + let path = CString::new(dir.path().join("nope.tgm").to_str().unwrap()).unwrap(); + + assert!(matches!( + super::tgm_streaming_encoder_create_with_encode_options( + ptr::null(), + meta.as_ptr(), + 0, + ptr::null(), + &mut enc + ), + super::TgmError::InvalidArg + )); + assert!(matches!( + super::tgm_streaming_encoder_create_with_encode_options( + path.as_ptr(), + ptr::null(), + 0, + ptr::null(), + &mut enc + ), + super::TgmError::InvalidArg + )); + assert!(matches!( + super::tgm_streaming_encoder_create_with_encode_options( + path.as_ptr(), + meta.as_ptr(), + 0, + ptr::null(), + ptr::null_mut() + ), + super::TgmError::InvalidArg + )); + assert!(enc.is_null()); + } } /// Compute a hash of the given data. diff --git a/rust/tensogram-ffi/tensogram.h b/rust/tensogram-ffi/tensogram.h index 102d5f70..5b04163c 100644 --- a/rust/tensogram-ffi/tensogram.h +++ b/rust/tensogram-ffi/tensogram.h @@ -74,6 +74,119 @@ typedef enum { TGM_ERROR_CANCELLED = 13, } tgm_error; +/** + * Where to place the aggregate hash frame (cbindgen: + * `tgm_aggregate_hash_policy`, variants `TGM_AGGREGATE_HASH_POLICY_*`). + * + * Mirrors [`tensogram::AggregateHashPolicy`]. `AUTO` is the zero value, so a + * zero-initialised [`TgmEncodeOptions`] asks the encoder to choose + * (`HEADER` when buffering, `FOOTER` when streaming) — the same default the + * Rust API has. + * + * `HEADER` and `BOTH` are **buffered-mode only**: a streaming encoder writes + * its header before any data object exists, so the per-object hashes are not + * yet known. [`tgm_streaming_encoder_create_with_encode_options`] rejects + * them with `TGM_ERROR_ENCODING` and an explanatory + * [`tgm_last_error`] message. + */ +typedef enum { + /** + * Encoder picks: buffered → header, streaming → footer. + */ + TGM_AGGREGATE_HASH_POLICY_AUTO = 0, + /** + * Emit no aggregate hash frame. Per-frame inline hash slots are + * unaffected. + */ + TGM_AGGREGATE_HASH_POLICY_NONE = 1, + /** + * Emit a `TGM_FRAME_TYPE_HEADER_HASH` frame. Buffered mode only. + */ + TGM_AGGREGATE_HASH_POLICY_HEADER = 2, + /** + * Emit a `TGM_FRAME_TYPE_FOOTER_HASH` frame. Valid in both modes. + */ + TGM_AGGREGATE_HASH_POLICY_FOOTER = 3, + /** + * Emit both a header and a footer hash frame carrying identical hash + * lists. Buffered mode only. + */ + TGM_AGGREGATE_HASH_POLICY_BOTH = 4, +} tgm_aggregate_hash_policy; + +/** + * Which codec implementation to use where both are compiled in (cbindgen: + * `tgm_compression_backend`, variants `TGM_COMPRESSION_BACKEND_*`). + * + * Mirrors [`tensogram::CompressionBackend`]. `AUTO` is the zero value: + * consult `TENSOGRAM_COMPRESSION_BACKEND`, else the platform default (FFI on + * native, pure-Rust on wasm32). `FFI` and `PURE` are explicit overrides that + * always win over the environment. + */ +typedef enum { + /** + * Consult the environment, then the platform default. + */ + TGM_COMPRESSION_BACKEND_AUTO = 0, + /** + * Use the C FFI codecs (libaec for szip, libzstd for zstd). + */ + TGM_COMPRESSION_BACKEND_FFI = 1, + /** + * Use the pure-Rust codecs (tensogram-szip, ruzstd). + */ + TGM_COMPRESSION_BACKEND_PURE = 2, +} tgm_compression_backend; + +/** + * An element type (cbindgen: `tgm_dtype`, variants `TGM_DTYPE_*`). + * + * Mirrors [`tensogram::Dtype`] one-for-one, numbered densely from `0` in core + * declaration order. The wire format stores the dtype as a **string** (see + * `plans/WIRE_FORMAT.md` §6.1), so these codes are an FFI convenience, not a + * wire value — but they are part of the C ABI and therefore frozen. + * [`tgm_object_dtype`] returns the same information as a string; this enum is + * what you want in a `switch`. + * + * The `impl From` below is an exhaustive match, so a new core variant + * fails the build here rather than silently mapping to something wrong. + */ +typedef enum { + TGM_DTYPE_FLOAT16 = 0, + TGM_DTYPE_BFLOAT16 = 1, + TGM_DTYPE_FLOAT32 = 2, + TGM_DTYPE_FLOAT64 = 3, + TGM_DTYPE_COMPLEX64 = 4, + TGM_DTYPE_COMPLEX128 = 5, + TGM_DTYPE_INT8 = 6, + TGM_DTYPE_INT16 = 7, + TGM_DTYPE_INT32 = 8, + TGM_DTYPE_INT64 = 9, + TGM_DTYPE_UINT8 = 10, + TGM_DTYPE_UINT16 = 11, + TGM_DTYPE_UINT32 = 12, + TGM_DTYPE_UINT64 = 13, + /** + * Sub-byte packed bitmask; element width is 1 bit, so the byte width of + * [`tensogram::Dtype::Bitmask`] is reported as 0. + */ + TGM_DTYPE_BITMASK = 14, +} tgm_dtype; + +/** + * A payload's byte order (cbindgen: `tgm_byte_order`, variants + * `TGM_BYTE_ORDER_*`). + * + * Mirrors [`tensogram::ByteOrder`]. Like [`TgmDtype`] these codes are an FFI + * convenience — the wire stores `"little"` / `"big"` as text — and are frozen + * as part of the C ABI. [`tgm_object_byte_order`] returns the same + * information as a string. + */ +typedef enum { + TGM_BYTE_ORDER_LITTLE = 0, + TGM_BYTE_ORDER_BIG = 1, +} tgm_byte_order; + /** * Which non-finite bitmask kind to fetch from a mask-aware decode. * @@ -116,6 +229,53 @@ typedef enum { TGM_VALUE_TYPE_MAP, } tgm_value_type; +/** + * A frame's type identifier (cbindgen: `tgm_frame_type`, variants + * `TGM_FRAME_TYPE_*`). + * + * Mirrors [`tensogram::FrameType`] value-for-value — these numbers are the + * wire's frame-type field (see `plans/WIRE_FORMAT.md` §2.2), not an FFI + * invention. **Type 4 is reserved** (it held the obsolete v2 data-object + * layout) and therefore has no variant, which is why the sequence skips from + * 3 to 5. `fortran/test/check_frame_type_enum.sh` guards the generated header + * against the core enum so the two can never drift. + */ +typedef enum { + /** + * CBOR global metadata, written in the header (random-access mode). + */ + TGM_FRAME_TYPE_HEADER_METADATA = 1, + /** + * Object index, written in the header (random-access mode). + */ + TGM_FRAME_TYPE_HEADER_INDEX = 2, + /** + * Aggregate hash frame, written in the header. + */ + TGM_FRAME_TYPE_HEADER_HASH = 3, + /** + * Aggregate hash frame, written in the footer. + */ + TGM_FRAME_TYPE_FOOTER_HASH = 5, + /** + * Object index, written in the footer (streaming mode). + */ + TGM_FRAME_TYPE_FOOTER_INDEX = 6, + /** + * CBOR global metadata, written in the footer (streaming mode). + */ + TGM_FRAME_TYPE_FOOTER_METADATA = 7, + /** + * Per-object metadata frame immediately preceding a data-object frame. + */ + TGM_FRAME_TYPE_PRECEDER_METADATA = 8, + /** + * N-dimensional tensor data-object frame — the only data-object type in + * v3, and the only frame type with a 20-byte footer. + */ + TGM_FRAME_TYPE_NTENSOR = 9, +} tgm_frame_type; + /** * Status returned by `tgm_async_streaming_encoder_try_object_count`. * @@ -188,6 +348,23 @@ typedef struct tgm_file_t tgm_file_t; */ typedef struct tgm_file_iter_t tgm_file_iter_t; +/** + * Opaque lazy cursor over one message's frames (cbindgen: + * `tgm_frame_iter_t`). Created by [`tgm_frame_iter_create`], advanced with + * [`tgm_frame_iter_next`], released with [`tgm_frame_iter_free`]. + * + * # Soundness invariant + * + * The handle stores the caller's message slice with a `'static` lifetime, + * which is a controlled fiction — exactly like the metadata value cursor's + * arena (see [`store_value`]). The C contract that makes it sound is stated + * on [`tgm_frame_iter_create`]: **`msg` must outlive the iterator**. Nothing + * inside the handle owns or copies the message bytes, so freeing the handle + * touches only the cursor; the caller's buffer is never read after + * [`tgm_frame_iter_free`] returns. + */ +typedef struct tgm_frame_iter_t tgm_frame_iter_t; + /** * Decoded message: global metadata + decoded (descriptor, payload) pairs. */ @@ -259,6 +436,74 @@ typedef struct { ptrdiff_t small_mask_threshold_bytes; } TgmEncodeMaskOptions; +/** + * The full encode-side option set (cbindgen: `TgmEncodeOptions`). + * + * Supersedes [`TgmEncodeMaskOptions`], which stays for source compatibility: + * it carries the same six mask fields **plus** the three knobs that + * previously had no C surface at all — the hash algorithm, where the + * aggregate hash frame goes, and which codec backend to use. + * + * Pass `NULL` to any `*_with_encode_options` entry point for the library + * defaults; a zero-initialised struct means the same thing, because every + * field's zero value *is* the default: + * + * | field | `NULL` / zero | meaning | + * |---|---|---| + * | `hash` | NULL | no hashing (the FFI convention: name it to get it) | + * | `aggregate_hash` | `TGM_AGGREGATE_HASH_POLICY_AUTO` | encoder picks the placement | + * | `compression_backend` | `TGM_COMPRESSION_BACKEND_AUTO` | env, then platform default | + * | `allow_nan` / `allow_inf` | `false` | non-finite input is a hard error | + * | `*_mask_method` | NULL | the library default (`"roaring"`) | + * | `small_mask_threshold_bytes` | negative | the library default (128) | + * + * `hash` is `"xxh3"` (v3's only algorithm), `"none"`, or NULL; anything else + * is [`TgmError::InvalidArg`]. Each `*_mask_method` is one of `"none"`, + * `"rle"`, `"roaring"`, `"lz4"`, `"zstd"`, `"blosc2"`. + * `small_mask_threshold_bytes` is the byte count below which mask blobs are + * stored raw regardless of the requested method; `0` disables that + * auto-fallback, negative values select the library default. + */ +typedef struct { + /** + * Hash algorithm name (`"xxh3"`, `"none"`) or NULL for no hashing. + */ + const char *hash; + /** + * Where to write the aggregate hash frame. Ignored when `hash` is NULL + * or `"none"` — there is nothing to aggregate. + */ + tgm_aggregate_hash_policy aggregate_hash; + /** + * Which codec implementation to prefer for szip / zstd. + */ + tgm_compression_backend compression_backend; + /** + * Substitute NaN with `0.0` and record a bitmask companion frame. + */ + bool allow_nan; + /** + * Substitute `±Inf` with `0.0` and record per-sign bitmask companions. + */ + bool allow_inf; + /** + * Compression method for the NaN mask, or NULL for the default. + */ + const char *nan_mask_method; + /** + * Compression method for the `+Inf` mask, or NULL for the default. + */ + const char *pos_inf_mask_method; + /** + * Compression method for the `-Inf` mask, or NULL for the default. + */ + const char *neg_inf_mask_method; + /** + * Raw-storage threshold for mask blobs; `0` disables, negative = default. + */ + ptrdiff_t small_mask_threshold_bytes; +} TgmEncodeOptions; + /** * Decode-side companion to [`TgmEncodeMaskOptions`]. Pass a pointer * to opt out of canonical NaN / Inf restoration. Pass `NULL` for @@ -300,6 +545,133 @@ typedef struct { uint64_t max_message_size; } TgmScanOptions; +/** + * Reader-side scan-walker options for [`tgm_file_open_remote`]. Mirrors + * [`tensogram::RemoteScanOptions`] as a flat, C-visible POD so callers can + * build it on the stack. + * + * Pass a `NULL` `TgmRemoteScanOptions*` to use the library defaults + * (`bidirectional = true`). + */ +typedef struct { + /** + * Enable the meet-in-the-middle (bidirectional) remote walk, which + * pairs forward preamble fetches with backward postamble fetches and + * roughly halves wall-clock layout discovery on real networks. `false` + * forces a forward-only walk. + */ + bool bidirectional; +} TgmRemoteScanOptions; + +/** + * One frame's structural description plus a **borrowed** view of its content. + * + * Filled by [`tgm_frame_iter_next`] into caller-provided storage. + * + * # Lifetime — read this before storing a `TgmFrame` + * + * `payload` points **into the `msg` buffer the caller passed to + * [`tgm_frame_iter_create`]** — it is a view, never a copy, and there is + * nothing to free. It stays valid for exactly as long as `msg` does, + * independently of any later `tgm_frame_iter_next` call and of + * `tgm_frame_iter_free`. Copy the bytes out if you need them to outlive the + * message buffer. + */ +typedef struct { + /** + * Which kind of frame this is. + */ + tgm_frame_type frame_type; + /** + * Frame-type-specific version field from the frame header. + */ + uint16_t version; + /** + * Raw 16-bit frame flags; bit 1 is `HASH_PRESENT` (see + * [`tgm_frame_has_hash`]). + */ + uint16_t flags; + /** + * Byte offset of the frame header, relative to the start of `msg`. + */ + size_t offset; + /** + * Whole-frame span in bytes: frame header through `ENDF`, excluding any + * alignment padding that follows. + */ + size_t length; + /** + * Borrowed content bytes: everything between the 16-byte frame header and + * the type-specific footer (20 bytes for `TGM_FRAME_TYPE_NTENSOR`, 12 for + * every other type). Points into the caller's `msg` — never freed. + */ + const uint8_t *payload; + /** + * Length of `payload` in bytes. + */ + size_t payload_len; +} TgmFrame; + +/** + * A message's envelope (the 24-byte preamble) as a flat C POD. + * + * Filled by [`tgm_message_header`]. The eight `has_*` members are the + * preamble's structural flags decoded into named booleans, so callers never + * touch a raw bitset. Together they say whether a message is *random-access* + * (metadata / index / hashes in the **header**) or *streaming* (in the + * **footer**) without reading a single frame. + * + * `total_length` is the whole-message byte count, preamble through + * postamble. It is `0` in a streaming message whose length was never + * back-filled — not an error, just "unknown at write time". + */ +typedef struct { + /** + * Wire-format version (`TGM_WIRE_VERSION` for messages this build writes). + */ + uint16_t version; + /** + * Total message length in bytes, or `0` if a streaming writer never + * back-filled it. + */ + uint64_t total_length; + /** + * A `HeaderMetadata` frame is present (random-access mode). + */ + bool has_header_metadata; + /** + * A `FooterMetadata` frame is present (streaming mode). + */ + bool has_footer_metadata; + /** + * A `HeaderIndex` frame is present. + */ + bool has_header_index; + /** + * A `FooterIndex` frame is present. + */ + bool has_footer_index; + /** + * A `HeaderHash` frame is present. + */ + bool has_header_hashes; + /** + * A `FooterHash` frame is present. + */ + bool has_footer_hashes; + /** + * At least one `PrecederMetadata` frame appears in the body. Advisory in + * streaming mode: the encoder sets it before it knows whether any + * preceder will be written, so `true` does not guarantee a frame. + */ + bool has_preceder_metadata; + /** + * Advisory: every frame in this message has its per-frame `HASH_PRESENT` + * bit set. For any single frame, `tgm_frame_has_hash` stays authoritative. + */ + bool has_hashes_present; +} TgmMessageHeader; + /** * Returns a pointer to the last error message, or NULL if no error. * The pointer is valid until the next FFI call on the same thread. @@ -368,6 +740,27 @@ tgm_error tgm_encode_with_options(const char *metadata_json, const TgmEncodeMaskOptions *mask_options, tgm_bytes_t *out); +/** + * Encode with the full [`TgmEncodeOptions`] set. + * + * Like [`tgm_encode_with_options`], but the option struct also carries the + * hash algorithm, the aggregate-hash placement and the compression backend — + * so there is no separate `hash_algo` argument here; put the algorithm name + * in `options->hash`. `NULL` options behave exactly like [`tgm_encode`] with + * a NULL `hash_algo`: no hashing, `AUTO` placement, `AUTO` backend, + * non-finite input rejected. + * + * On success returns `TGM_ERROR_OK` and fills `out` with the encoded bytes, + * which the caller frees with [`tgm_bytes_free`]. + */ +tgm_error tgm_encode_with_encode_options(const char *metadata_json, + const uint8_t *const *data_ptrs, + const size_t *data_lens, + size_t num_objects, + uint32_t threads, + const TgmEncodeOptions *options, + tgm_bytes_t *out); + /** * Decode with explicit NaN / Inf restoration options. * @@ -397,6 +790,26 @@ tgm_error tgm_streaming_encoder_create_with_options(const char *path, const TgmEncodeMaskOptions *mask_options, tgm_streaming_encoder_t **out); +/** + * Streaming-encoder constructor taking the full [`TgmEncodeOptions`] set. + * + * Like [`tgm_streaming_encoder_create_with_options`], but the option struct + * also carries the hash algorithm, the aggregate-hash placement and the + * compression backend, so there is no separate `hash_algo` argument. `NULL` + * options behave like [`tgm_streaming_encoder_create`] with a NULL + * `hash_algo`. + * + * `TGM_AGGREGATE_HASH_POLICY_HEADER` and `..._BOTH` are rejected here with + * `TGM_ERROR_ENCODING`: a streaming writer emits its header before any data + * object exists, so the per-object hashes are not yet known. Use `AUTO` + * (which resolves to the footer when streaming) or `FOOTER`. + */ +tgm_error tgm_streaming_encoder_create_with_encode_options(const char *path, + const char *metadata_json, + uint32_t threads, + const TgmEncodeOptions *options, + tgm_streaming_encoder_t **out); + /** * Append a message to a file with explicit NaN / Inf mask-companion options. * @@ -412,6 +825,22 @@ tgm_error tgm_file_append_with_options(tgm_file_t *file, uint32_t threads, const TgmEncodeMaskOptions *mask_options); +/** + * Append a message to a file with the full [`TgmEncodeOptions`] set. + * + * Like [`tgm_file_append_with_options`], but the option struct also carries + * the hash algorithm, the aggregate-hash placement and the compression + * backend, so there is no separate `hash_algo` argument. `NULL` options + * behave like [`tgm_file_append`] with a NULL `hash_algo`. + */ +tgm_error tgm_file_append_with_encode_options(tgm_file_t *file, + const char *metadata_json, + const uint8_t *const *data_ptrs, + const size_t *data_lens, + size_t num_objects, + uint32_t threads, + const TgmEncodeOptions *options); + /** * Encode a Tensogram message from JSON metadata and pre-encoded payload bytes. * @@ -667,6 +1096,20 @@ const uint64_t *tgm_object_strides(const tgm_message_t *msg, size_t index); */ const char *tgm_object_dtype(const tgm_message_t *msg, size_t index); +/** + * Returns the object's dtype as a [`tgm_dtype`](TgmDtype) code — the typed + * companion to [`tgm_object_dtype`], for callers that want to `switch` + * instead of `strcmp`. + * + * A NULL `msg` or an out-of-range `index` records the reason in + * [`tgm_last_error`] and returns the zero-valued variant + * (`TGM_DTYPE_FLOAT16`); an enum return has no spare code to signal + * failure. To bounds-check unambiguously, compare `index` against + * [`tgm_message_num_objects`], or call [`tgm_object_dtype`], which returns + * NULL for exactly the same inputs. + */ +tgm_dtype tgm_object_dtype_enum(const tgm_message_t *msg, size_t index); + /** * Returns a pointer to the decoded payload bytes for a decoded object. * `decoded_index` is the index into the decoded objects array (0 for the @@ -721,6 +1164,17 @@ const char *tgm_object_type(const tgm_message_t *msg, size_t index); */ const char *tgm_object_byte_order(const tgm_message_t *msg, size_t index); +/** + * Returns the object's byte order as a [`tgm_byte_order`](TgmByteOrder) code + * — the typed companion to [`tgm_object_byte_order`]. + * + * A NULL `msg` or an out-of-range `index` records the reason in + * [`tgm_last_error`] and returns the zero-valued variant + * (`TGM_BYTE_ORDER_LITTLE`); see [`tgm_object_dtype_enum`] for how to + * bounds-check unambiguously. + */ +tgm_byte_order tgm_object_byte_order_enum(const tgm_message_t *msg, size_t index); + /** * Returns the filter string (e.g. "none", "shuffle"). Valid until message freed. */ @@ -1171,6 +1625,53 @@ tgm_error tgm_file_append(tgm_file_t *file, */ void tgm_file_close(tgm_file_t *file); +/** + * `true` when `source` is a URL this build can open remotely. + * + * Mirrors [`tensogram::is_remote_url`]: the recognised schemes are `s3`, + * `s3a`, `gs`, `az`, `azure`, `http` and `https`, compared + * case-insensitively. Plain paths and `file://` URLs are **not** remote — + * they belong to the local backend ([`tgm_file_open`]). + * + * Returns `false` — with the reason in [`tgm_last_error`] — for a NULL or + * non-UTF-8 `source`, and for **every** input when this build was compiled + * without the `remote` Cargo feature (such a build genuinely cannot open any + * remote URL, so "not remote for me" is the honest answer; the error message + * says how to fix it). + */ +bool tgm_is_remote_url(const char *source); + +/** + * Open a remote `.tgm` (S3 / GCS / Azure / HTTP) for **synchronous** reading. + * + * The blocking counterpart to `tgm_async_file_open_remote`: on success `*out` + * receives an ordinary [`tgm_file_t`](TgmFile), so the whole existing file + * API — [`tgm_file_message_count`], [`tgm_file_read_message`], + * [`tgm_file_decode_message`], [`tgm_file_iter_create`], … — works unchanged + * against the remote source. Close it with [`tgm_file_close`] as usual. + * + * `keys` / `values` are parallel arrays of `n_options` backend storage + * options (credentials, region, endpoint, …) forwarded verbatim to the + * object-store backend; pass `0` / NULL for none. `opts` configures the scan + * walker; NULL selects the library defaults (`bidirectional = true`). + * + * Argument validation (NULL `source` / `out`, malformed option arrays) runs + * **before** the feature check, so every build answers + * `TGM_ERROR_INVALID_ARG` to the same mistakes. Remote failures — an + * unparseable URL, a missing object, a rejected storage option, transport + * errors — map to `TGM_ERROR_REMOTE` with the detail in [`tgm_last_error`]. + * + * Always exported so consumers linking the cdylib never see an undefined + * symbol; a build without the `remote` Cargo feature returns + * `TGM_ERROR_REMOTE` and explains how to enable it. + */ +tgm_error tgm_file_open_remote(const char *source, + const char *const *keys, + const char *const *values, + size_t n_options, + const TgmRemoteScanOptions *opts, + tgm_file_t **out); + /** * Compute simple_packing parameters for a set of f64 values. * @@ -1304,6 +1805,84 @@ tgm_error tgm_object_iter_next(tgm_object_iter_t *iter, tgm_message_t **out); */ void tgm_object_iter_free(tgm_object_iter_t *iter); +/** + * Start a lazy walk over the frames of one message. + * + * `msg` must point at the start of a message (the `TENSOGRM` preamble magic) + * — typically a slice obtained from `tgm_scan`. Only the type 1–9 `FR` frames + * are yielded; the preamble and postamble are not frames, use + * [`tgm_message_header`] for the envelope. + * + * Returns NULL — with the reason in `tgm_last_error` — if `msg` is NULL or + * the preamble does not parse (truncated buffer, wrong magic, unsupported + * version). Free the returned handle with [`tgm_frame_iter_free`]. + * + * # Lifetime contract + * + * The iterator **borrows `msg`; `msg` must outlive the iterator** and must + * not be moved, reallocated, or mutated while the iterator lives. Each frame + * written by [`tgm_frame_iter_next`] carries a `payload` pointer **into + * `msg`**, which remains valid for as long as `msg` lives — later `next` + * calls and `tgm_frame_iter_free` do not invalidate it. + * + * Binds [`tensogram::frames`]. + */ +tgm_frame_iter_t *tgm_frame_iter_create(const uint8_t *msg, size_t msg_len); + +/** + * Advance the frame cursor, filling `*out` with the next frame. + * + * Returns `true` when a frame was written. Returns `false` in three cases, + * which `tgm_last_error` tells apart: + * + * - **clean end** — every frame has been yielded; the last error is + * *cleared*, so `tgm_last_error()` returns NULL; + * - **malformed frame** — the frame chain is truncated or inconsistent; the + * reason is recorded in `tgm_last_error()` and iteration stops for good; + * - **invalid argument** — `it` or `out` is NULL; nothing is written. + * + * Calling this again after any `false` is safe and keeps returning `false`. + * + * `out->payload` borrows the caller's message buffer — see the lifetime + * contract on [`tgm_frame_iter_create`]. + */ +bool tgm_frame_iter_next(tgm_frame_iter_t *it, TgmFrame *out); + +/** + * Free a frame cursor. Releases only the cursor — the caller's message + * buffer and every `payload` pointer handed out by [`tgm_frame_iter_next`] + * are untouched and stay valid. NULL is a no-op. + */ +void tgm_frame_iter_free(tgm_frame_iter_t *it); + +/** + * `true` if this frame's `HASH_PRESENT` flag is set, i.e. its inline hash + * slot holds a meaningful digest (see `plans/WIRE_FORMAT.md` §2.5). + * + * Convenience over `frame->flags` bit 1, and the authoritative answer for a + * single frame — `TgmMessageHeader::has_hashes_present` is only an advisory + * message-wide summary. Returns `false` for a NULL `frame`. + * + * Binds [`tensogram::FrameInfo::has_hash`]. + */ +bool tgm_frame_has_hash(const TgmFrame *frame); + +/** + * Read a message's envelope without walking its frames. + * + * `msg` must point at the start of a message (the `TENSOGRM` preamble magic) + * — typically a slice obtained from `tgm_scan`. On success writes the decoded + * preamble to `*out` and returns `TGM_ERROR_OK`. + * + * Returns `TGM_ERROR_INVALID_ARG` if `msg` or `out` is NULL, or the mapped + * error code if the preamble does not parse (message truncated, wrong magic, + * unsupported version); the reason is available from `tgm_last_error` and + * `*out` is left untouched. + * + * Binds [`tensogram::message_header`]. + */ +tgm_error tgm_message_header(const uint8_t *msg, size_t msg_len, TgmMessageHeader *out); + /** * Convert an error code to a human-readable string. * Returns a static string (always valid, never NULL). diff --git a/rust/tensogram-wasm/src/convert.rs b/rust/tensogram-wasm/src/convert.rs index 598066de..ef1b849c 100644 --- a/rust/tensogram-wasm/src/convert.rs +++ b/rust/tensogram-wasm/src/convert.rs @@ -12,12 +12,12 @@ //! `wasm_bindgen::memory()` + byte offsets for zero-copy TypedArray //! views that avoid forming misaligned Rust references. //! -//! Also hosts the shared helpers (`js_err`, `build_encode_options`, -//! `extract_descriptor_data_pairs`) used by every WASM entrypoint -//! that writes a message. +//! Also hosts the shared helpers (`js_err`, `set_field`, +//! `build_encode_options`, `extract_descriptor_data_pairs`) used by +//! every WASM entrypoint that writes a message. use serde::Serialize; -use tensogram::{self as core, EncodeOptions}; +use tensogram::{self as core, AggregateHashPolicy, EncodeOptions}; use wasm_bindgen::prelude::*; /// Convert a [`tensogram::TensogramError`] to a thrown JS error @@ -93,6 +93,14 @@ fn attach_string_prop(err: &JsValue, key: &str, value: &str) { let _ = js_sys::Reflect::set(err, &key.into(), &value.into()); } +/// Set an own-property on a JS object, mapping the (practically +/// unreachable) `Reflect::set` failure to a thrown JS error. +pub(crate) fn set_field(obj: &js_sys::Object, key: &str, value: &JsValue) -> Result<(), JsValue> { + js_sys::Reflect::set(obj, &JsValue::from_str(key), value) + .map(|_| ()) + .map_err(|_| JsValue::from(js_sys::Error::new(&format!("failed to set '{key}'")))) +} + /// Best-effort attach a numeric (`usize`-valued) own-property. fn attach_number_prop(err: &JsValue, key: &str, value: usize) { // JS Number is f64; up to 2^53 - 1 is exactly representable. @@ -109,10 +117,32 @@ fn attach_number_prop(err: &JsValue, key: &str, value: usize) { /// [`build_encode_options_full`] directly. Infallible — this form /// cannot carry user-supplied mask-method names. pub(crate) fn build_encode_options(hash: Option) -> EncodeOptions { - build_encode_options_full(hash, None, None, None, None, None, None) + build_encode_options_full(hash, None, None, None, None, None, None, None) .expect("build_encode_options_full with no method names cannot error") } +/// Parse the JS `aggregateHash` option name into an +/// [`AggregateHashPolicy`]. +/// +/// `None` (option omitted) means [`AggregateHashPolicy::Auto`], which +/// the buffered encoder resolves to a `HeaderHash` frame and the +/// streaming encoder to a `FooterHash` frame. Unknown names are +/// rejected with the full list of accepted values — no silent +/// fallback, matching the mask-method contract above. +fn parse_aggregate_hash(name: Option<&str>) -> Result { + match name { + None | Some("auto") => Ok(AggregateHashPolicy::Auto), + Some("none") => Ok(AggregateHashPolicy::None), + Some("header") => Ok(AggregateHashPolicy::Header), + Some("footer") => Ok(AggregateHashPolicy::Footer), + Some("both") => Ok(AggregateHashPolicy::Both), + Some(other) => Err(JsValue::from(js_sys::Error::new(&format!( + "unknown aggregateHash policy '{other}', expected one of: \ + auto, none, header, footer, both" + )))), + } +} + /// Build an [`EncodeOptions`] from the full JS kwargs set for the /// NaN / Inf bitmask companion frame. See /// `docs/src/guide/nan-inf-handling.md` for the semantics and @@ -126,6 +156,10 @@ pub(crate) fn build_encode_options(hash: Option) -> EncodeOptions { /// names the offending value and the full list of accepted names /// — no silent fallback. /// - `small_mask_threshold_bytes`: default `128`. +/// - `aggregate_hash`: optional policy name (`"auto"` | `"none"` | +/// `"header"` | `"footer"` | `"both"`) selecting where the aggregate +/// hash frame is written. `None` means `"auto"`. See +/// [`parse_aggregate_hash`]. #[allow(clippy::too_many_arguments)] pub(crate) fn build_encode_options_full( hash: Option, @@ -135,6 +169,7 @@ pub(crate) fn build_encode_options_full( pos_inf_mask_method: Option<&str>, neg_inf_mask_method: Option<&str>, small_mask_threshold_bytes: Option, + aggregate_hash: Option<&str>, ) -> Result { use core::encode::MaskMethod; @@ -154,6 +189,7 @@ pub(crate) fn build_encode_options_full( neg_inf_mask_method: parse(neg_inf_mask_method, defaults.neg_inf_mask_method.clone())?, small_mask_threshold_bytes: small_mask_threshold_bytes .unwrap_or(defaults.small_mask_threshold_bytes), + aggregate_hash: parse_aggregate_hash(aggregate_hash)?, ..defaults }) } diff --git a/rust/tensogram-wasm/src/encoder.rs b/rust/tensogram-wasm/src/encoder.rs index c9558351..ea7c27b9 100644 --- a/rust/tensogram-wasm/src/encoder.rs +++ b/rust/tensogram-wasm/src/encoder.rs @@ -220,6 +220,13 @@ impl StreamingEncoder { pos_inf_mask_method.as_deref(), neg_inf_mask_method.as_deref(), small_mask_threshold_bytes, + // The streaming encoder keeps the core default + // (`AggregateHashPolicy::Auto` → a `FooterHash` frame): + // header placement is impossible here because the header is + // written before any object, so the digests are not yet + // known. Buffered `encode` carries the caller-selectable + // knob. + None, )?; let inner = match on_bytes { Some(cb) => { diff --git a/rust/tensogram-wasm/src/extras.rs b/rust/tensogram-wasm/src/extras.rs index 261ac981..abb9d2a7 100644 --- a/rust/tensogram-wasm/src/extras.rs +++ b/rust/tensogram-wasm/src/extras.rs @@ -44,14 +44,6 @@ pub fn decode_descriptors(buf: &[u8]) -> Result { Ok(result.into()) } -/// Set an own-property on a JS object, mapping the (practically -/// unreachable) `Reflect::set` failure to a thrown JS error. -fn set_field(obj: &js_sys::Object, key: &str, value: &JsValue) -> Result<(), JsValue> { - js_sys::Reflect::set(obj, &JsValue::from_str(key), value) - .map(|_| ()) - .map_err(|_| JsValue::from(js_sys::Error::new(&format!("failed to set '{key}'")))) -} - // ── scan_with_options ──────────────────────────────────────────────────────── /// Scan a multi-message buffer for message boundaries with explicit diff --git a/rust/tensogram-wasm/src/frame_walk.rs b/rust/tensogram-wasm/src/frame_walk.rs new file mode 100644 index 00000000..85f1df4e --- /dev/null +++ b/rust/tensogram-wasm/src/frame_walk.rs @@ -0,0 +1,175 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +//! WASM bindings for the structural frame walker. +//! +//! Mirrors the Rust core [`tensogram::frames`] / +//! [`tensogram::message_header`] (`rust/tensogram/src/frame_walk.rs`): +//! inspect one message's *shape* — which frames it holds and where — +//! without decoding payloads or CBOR. +//! +//! # One message per call +//! +//! Both entry points take the bytes of a **single** message, starting +//! at its `TENSOGRM` preamble magic. Callers holding a multi-message +//! buffer must use [`crate::scan`] to obtain `[offset, length]` pairs +//! and slice before calling. +//! +//! # Payload ownership +//! +//! [`frames`] returns each frame's content as a JS-heap `Uint8Array` +//! **copy** (via `Uint8Array::from`), matching the mask accessors on +//! [`crate::DecodedMaskedMessage`]. A zero-copy view is not merely +//! undesirable here, it is unsound: `wasm_bindgen` materialises the +//! `&[u8]` argument in a temporary WASM allocation that is released +//! when this function returns, so a view into the walked payload would +//! dangle the moment the call completes. Copies are also immune to +//! WASM linear-memory growth, so callers may retain them freely. +//! +//! # Eager materialisation +//! +//! The core walker is a lazy iterator; this binding materialises it +//! into a JS array because the frames-per-message count is small +//! (header frames + one per data object) and a lazily-pulled iterator +//! would have to keep the message bytes alive across the WASM +//! boundary. A malformed frame chain therefore surfaces as a thrown +//! error rather than a truncated array — corruption is never silently +//! rendered as "fewer frames". + +use crate::convert::{js_err, set_field, to_js}; +use serde::Serialize; +use tensogram::{self as core, wire::FrameType}; +use wasm_bindgen::prelude::*; + +/// The canonical wire-format name of a frame type, as spelled in +/// `plans/WIRE_FORMAT.md` §6 and in the Rust `FrameType` enum. Kept as +/// an explicit match so a new frame type cannot be added to the core +/// without this binding failing to compile. +fn frame_type_name(ft: FrameType) -> &'static str { + match ft { + FrameType::HeaderMetadata => "HeaderMetadata", + FrameType::HeaderIndex => "HeaderIndex", + FrameType::HeaderHash => "HeaderHash", + FrameType::FooterHash => "FooterHash", + FrameType::FooterIndex => "FooterIndex", + FrameType::FooterMetadata => "FooterMetadata", + FrameType::PrecederMetadata => "PrecederMetadata", + FrameType::NTensorFrame => "NTensorFrame", + } +} + +/// Walk the frames of a single Tensogram message. +/// +/// Thin binding over [`tensogram::frames`]. The preamble and +/// postamble are **not** frames and are not returned — use +/// [`message_header`] for the envelope. +/// +/// @param message - Wire-format bytes of ONE message, starting at the +/// `TENSOGRM` preamble magic. Use [`crate::scan`] to locate messages +/// in a multi-message buffer and slice before calling. +/// @returns An array of plain JS objects in wire order, one per frame: +/// `{ frame_type, frame_type_code, version, flags, offset, length, +/// payload, has_hash }`. +/// - `frame_type` — canonical name (`"HeaderMetadata"` … +/// `"NTensorFrame"`); `frame_type_code` — the wire number (1–9). +/// - `offset` — byte offset of the frame header, relative to the +/// start of `message`; `length` — whole-frame span through `ENDF`, +/// excluding trailing alignment padding. +/// - `payload` — the frame's content as a JS-heap `Uint8Array` copy: +/// the 16-byte frame header and the type-specific footer (20 B for +/// frame type 9, 12 B otherwise) are stripped. +/// - `has_hash` — the frame's `HASH_PRESENT` flag. +/// +/// Throws when `message` does not start with a valid preamble, or when +/// the frame chain is malformed (truncated frame header, a declared +/// `total_length` that does not fit). +#[wasm_bindgen] +pub fn frames(message: &[u8]) -> Result { + let out = js_sys::Array::new(); + for item in core::frames(message).map_err(js_err)? { + let info = item.map_err(js_err)?; + let obj = js_sys::Object::new(); + set_field( + &obj, + "frame_type", + &JsValue::from_str(frame_type_name(info.frame_type)), + )?; + set_field( + &obj, + "frame_type_code", + &JsValue::from(info.frame_type as u16), + )?; + set_field(&obj, "version", &JsValue::from(info.version))?; + set_field(&obj, "flags", &JsValue::from(info.flags))?; + // `offset` / `length` are `usize`, i.e. u32 on wasm32 — every + // value is exactly representable as a JS number. + set_field(&obj, "offset", &JsValue::from(info.offset as u32))?; + set_field(&obj, "length", &JsValue::from(info.length as u32))?; + set_field( + &obj, + "payload", + &js_sys::Uint8Array::from(info.payload).into(), + )?; + set_field(&obj, "has_hash", &JsValue::from_bool(info.has_hash()))?; + out.push(&obj); + } + Ok(out) +} + +/// The message envelope, as returned by [`message_header`]. +#[derive(Serialize)] +struct MessageHeaderJs { + version: u16, + total_length: u64, + has_header_metadata: bool, + has_footer_metadata: bool, + has_header_index: bool, + has_footer_index: bool, + has_header_hashes: bool, + has_footer_hashes: bool, + has_preceder_metadata: bool, + has_hashes_present: bool, +} + +/// Read one message's envelope (preamble) as typed values, without +/// walking its frames. +/// +/// Thin binding over [`tensogram::message_header`]. Tells a caller +/// whether a message is random-access (metadata / index / hashes in the +/// *header*) or streaming (in the *footer*) from 24 bytes alone. +/// +/// @param message - Wire-format bytes of ONE message, starting at the +/// `TENSOGRM` preamble magic. +/// @returns `{ version, total_length, has_header_metadata, +/// has_footer_metadata, has_header_index, has_footer_index, +/// has_header_hashes, has_footer_hashes, has_preceder_metadata, +/// has_hashes_present }`. `total_length` is `0` for a streaming +/// message whose length was never back-filled. +/// +/// The predicates are exact for a buffered (random-access) message — +/// the encoder knows the whole message up front. A streaming encoder +/// writes the preamble before any object, so its flags are advisory: +/// only `frame present ⇒ flag set` is guaranteed there. +/// +/// Throws when `message` does not start with a valid preamble. +#[wasm_bindgen] +pub fn message_header(message: &[u8]) -> Result { + let h = core::message_header(message).map_err(js_err)?; + to_js(&MessageHeaderJs { + version: h.version, + total_length: h.total_length, + has_header_metadata: h.has_header_metadata(), + has_footer_metadata: h.has_footer_metadata(), + has_header_index: h.has_header_index(), + has_footer_index: h.has_footer_index(), + has_header_hashes: h.has_header_hashes(), + has_footer_hashes: h.has_footer_hashes(), + has_preceder_metadata: h.has_preceder_metadata(), + has_hashes_present: h.has_hashes_present(), + }) +} diff --git a/rust/tensogram-wasm/src/lib.rs b/rust/tensogram-wasm/src/lib.rs index cb5666c7..c7227bfd 100644 --- a/rust/tensogram-wasm/src/lib.rs +++ b/rust/tensogram-wasm/src/lib.rs @@ -24,6 +24,7 @@ mod convert; mod encoder; mod extras; +mod frame_walk; mod layout; mod masks; mod remote_scan; @@ -151,6 +152,12 @@ pub fn scan(buf: &[u8]) -> Result { /// @param pos_inf_mask_method - Mask compression method for the +Inf mask /// @param neg_inf_mask_method - Mask compression method for the -Inf mask /// @param small_mask_threshold_bytes - Mask size below which method="none" is forced (default: 128) +/// @param aggregate_hash - Where to emit the aggregate hash frame: +/// `"auto"` (default — a `HeaderHash` frame in this +/// buffered path), `"none"`, `"header"`, `"footer"`, +/// or `"both"`. Ignored when `hash` is `false`: +/// with per-frame hashing off there is nothing to +/// aggregate. Unknown names throw. /// @returns Uint8Array containing the encoded .tgm message #[wasm_bindgen] #[allow(clippy::too_many_arguments)] @@ -164,6 +171,7 @@ pub fn encode( pos_inf_mask_method: Option, neg_inf_mask_method: Option, small_mask_threshold_bytes: Option, + aggregate_hash: Option, ) -> Result { let metadata = metadata_from_js(&metadata_js)?; let (descriptors, data_vec) = extract_descriptor_data_pairs(&objects_js)?; @@ -180,6 +188,7 @@ pub fn encode( pos_inf_mask_method.as_deref(), neg_inf_mask_method.as_deref(), small_mask_threshold_bytes, + aggregate_hash.as_deref(), )?; let encoded = core::encode(&metadata, &pairs, &options).map_err(js_err)?; // Return a JS-owned copy. We must not use `view_as_u8` here because @@ -328,6 +337,10 @@ pub use extras::{ pub use masks::{DecodedMaskedMessage, decode_with_masks}; +// ── Structural introspection (frame walker + typed message envelope) ──────── + +pub use frame_walk::{frames, message_header}; + // ── Doctor: environment diagnostics ────────────────────────────────────────── /// Collect environment diagnostics: build metadata, compiled-in feature diff --git a/rust/tensogram-wasm/tests/frame_walk_tests.rs b/rust/tensogram-wasm/tests/frame_walk_tests.rs new file mode 100644 index 00000000..58c50d08 --- /dev/null +++ b/rust/tensogram-wasm/tests/frame_walk_tests.rs @@ -0,0 +1,283 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +//! Tests for the frame-walker exports in +//! `rust/tensogram-wasm/src/frame_walk.rs`. +//! +//! These cover the WASM binding layer directly — the JS object shape it +//! emits, the payload-copy contract, and the `aggregate_hash` name +//! guard, which the TypeScript wrapper's own validation shadows and so +//! cannot reach. +//! +//! Run with: wasm-pack test --node rust/tensogram-wasm + +use std::collections::BTreeMap; +use tensogram::dtype::Dtype; +use tensogram::types::{ByteOrder, DataObjectDescriptor, GlobalMetadata}; +use tensogram::wire::{FRAME_HEADER_SIZE, POSTAMBLE_SIZE, PREAMBLE_SIZE}; +use tensogram_wasm::{encode, frames, message_header}; +use wasm_bindgen::JsCast; +use wasm_bindgen_test::*; + +fn descriptor(shape: Vec) -> DataObjectDescriptor { + let mut strides = vec![1u64; shape.len()]; + for i in (0..shape.len().saturating_sub(1)).rev() { + strides[i] = strides[i + 1] * shape[i + 1]; + } + DataObjectDescriptor { + obj_type: "ntensor".to_string(), + ndim: shape.len() as u64, + shape, + strides, + dtype: Dtype::Float32, + byte_order: ByteOrder::Little, + encoding: "none".to_string(), + filter: "none".to_string(), + compression: "none".to_string(), + params: BTreeMap::new(), + masks: None, + } +} + +/// A buffered (random-access) message holding `n` four-element f32 +/// objects, encoded with the supplied options. +fn message_with(n: usize, options: &tensogram::EncodeOptions) -> Vec { + let desc = descriptor(vec![4]); + let data: Vec = (0..4u32).flat_map(|i| (i as f32).to_le_bytes()).collect(); + let pairs: Vec<(&DataObjectDescriptor, &[u8])> = + (0..n).map(|_| (&desc, data.as_slice())).collect(); + tensogram::encode(&GlobalMetadata::default(), &pairs, options).unwrap() +} + +fn message(n: usize) -> Vec { + message_with(n, &tensogram::EncodeOptions::default()) +} + +fn frame_at(arr: &js_sys::Array, i: u32) -> js_sys::Object { + arr.get(i).dyn_into().expect("frame is a JS object") +} + +fn string_field(obj: &js_sys::Object, key: &str) -> String { + js_sys::Reflect::get(obj, &key.into()) + .unwrap() + .as_string() + .unwrap_or_else(|| panic!("{key} not a string")) +} + +fn number_field(obj: &js_sys::Object, key: &str) -> f64 { + js_sys::Reflect::get(obj, &key.into()) + .unwrap() + .as_f64() + .unwrap_or_else(|| panic!("{key} not a number")) +} + +fn bool_field(obj: &js_sys::Object, key: &str) -> bool { + js_sys::Reflect::get(obj, &key.into()) + .unwrap() + .as_bool() + .unwrap_or_else(|| panic!("{key} not a boolean")) +} + +fn payload_field(obj: &js_sys::Object) -> js_sys::Uint8Array { + js_sys::Reflect::get(obj, &"payload".into()) + .unwrap() + .dyn_into() + .expect("payload is a Uint8Array") +} + +fn frame_type_names(msg: &[u8]) -> Vec { + let arr = frames(msg).unwrap(); + (0..arr.length()) + .map(|i| string_field(&frame_at(&arr, i), "frame_type")) + .collect() +} + +// ── frames ─────────────────────────────────────────────────────────────────── + +#[wasm_bindgen_test] +fn frames_emit_canonical_wire_names_in_order() { + assert_eq!( + frame_type_names(&message(2)), + vec![ + "HeaderMetadata", + "HeaderIndex", + "HeaderHash", + "NTensorFrame", + "NTensorFrame", + ] + ); +} + +#[wasm_bindgen_test] +fn frame_objects_carry_every_documented_field() { + let msg = message(1); + let arr = frames(&msg).unwrap(); + for i in 0..arr.length() { + let f = frame_at(&arr, i); + let code = number_field(&f, "frame_type_code") as u16; + assert!((1..=9).contains(&code), "frame_type_code in the wire range"); + assert_eq!(number_field(&f, "version") as u16, 1); + let offset = number_field(&f, "offset") as usize; + let length = number_field(&f, "length") as usize; + assert!(offset >= PREAMBLE_SIZE, "the preamble is not a frame"); + assert!(offset + length <= msg.len() - POSTAMBLE_SIZE); + // has_hash mirrors bit 1 of the raw frame flags. + let flags = number_field(&f, "flags") as u16; + assert_eq!(bool_field(&f, "has_hash"), flags & (1 << 1) != 0); + // The payload excludes the frame header and the type footer. + let footer = if code == 9 { 20 } else { 12 }; + assert_eq!( + payload_field(&f).length() as usize, + length - FRAME_HEADER_SIZE - footer + ); + } +} + +#[wasm_bindgen_test] +fn frame_payload_is_a_js_heap_copy_not_a_view() { + // `wasm_bindgen` materialises the `&[u8]` argument in a temporary + // WASM allocation that is freed when `frames` returns, so a view + // would dangle. Round-tripping the bytes after further WASM + // allocations proves the copy is independent. + let msg = message(1); + let arr = frames(&msg).unwrap(); + let f = frame_at(&arr, 0); + let payload = payload_field(&f); + let snapshot = payload.to_vec(); + // Allocate hard in WASM linear memory; a view would be invalidated. + let _churn = frames(&message(64)).unwrap(); + assert_eq!(payload.to_vec(), snapshot); + let offset = number_field(&f, "offset") as usize; + let start = offset + FRAME_HEADER_SIZE; + assert_eq!(snapshot, msg[start..start + snapshot.len()].to_vec()); +} + +#[wasm_bindgen_test] +fn frames_reject_a_buffer_that_is_not_a_message() { + assert!(frames(b"not a tensogram message").is_err()); + assert!(frames(&[]).is_err()); + assert!(frames(&[0u8; 64]).is_err()); +} + +#[wasm_bindgen_test] +fn frames_reject_a_truncated_frame_chain() { + let msg = message(2); + let arr = frames(&msg).unwrap(); + let last = frame_at(&arr, arr.length() - 1); + let cut = number_field(&last, "offset") as usize + 8; + // The preamble still parses; the chain does not. The eager array + // must surface the error rather than return a shorter list. + assert!(frames(&msg[..cut]).is_err()); +} + +// ── message_header ─────────────────────────────────────────────────────────── + +#[wasm_bindgen_test] +fn message_header_flags_match_the_frames_present() { + let msg = message(2); + let header: js_sys::Object = message_header(&msg).unwrap().dyn_into().unwrap(); + assert_eq!(number_field(&header, "version") as u16, 3); + assert_eq!(number_field(&header, "total_length") as usize, msg.len()); + + let names = frame_type_names(&msg); + let has = |n: &str| names.iter().any(|t| t == n); + assert_eq!( + bool_field(&header, "has_header_metadata"), + has("HeaderMetadata") + ); + assert_eq!( + bool_field(&header, "has_footer_metadata"), + has("FooterMetadata") + ); + assert_eq!(bool_field(&header, "has_header_index"), has("HeaderIndex")); + assert_eq!(bool_field(&header, "has_footer_index"), has("FooterIndex")); + assert_eq!(bool_field(&header, "has_header_hashes"), has("HeaderHash")); + assert_eq!(bool_field(&header, "has_footer_hashes"), has("FooterHash")); + assert_eq!( + bool_field(&header, "has_preceder_metadata"), + has("PrecederMetadata") + ); + assert!(bool_field(&header, "has_hashes_present")); +} + +#[wasm_bindgen_test] +fn message_header_rejects_a_buffer_that_is_not_a_message() { + assert!(message_header(b"not a tensogram message").is_err()); + assert!(message_header(&[]).is_err()); +} + +// ── aggregate hash placement, observed through the walker ──────────────────── + +#[wasm_bindgen_test] +fn aggregate_hash_both_emits_a_header_and_a_footer_hash_frame() { + let options = tensogram::EncodeOptions { + aggregate_hash: tensogram::AggregateHashPolicy::Both, + ..Default::default() + }; + let names = frame_type_names(&message_with(2, &options)); + assert_eq!(names.iter().filter(|t| *t == "HeaderHash").count(), 1); + assert_eq!(names.iter().filter(|t| *t == "FooterHash").count(), 1); + let header_at = names.iter().position(|t| t == "HeaderHash").unwrap(); + let footer_at = names.iter().position(|t| t == "FooterHash").unwrap(); + let first_object = names.iter().position(|t| t == "NTensorFrame").unwrap(); + let last_object = names.iter().rposition(|t| t == "NTensorFrame").unwrap(); + assert!( + header_at < first_object, + "header aggregate precedes the body" + ); + assert!(footer_at > last_object, "footer aggregate follows the body"); +} + +/// Call the JS-facing `encode` with no objects and the given +/// `aggregate_hash` name — enough input to reach the option parser. +fn encode_with_aggregate_hash( + name: Option<&str>, +) -> Result { + encode( + js_sys::Object::new().into(), + js_sys::Array::new(), + None, + None, + None, + None, + None, + None, + None, + name.map(str::to_string), + ) +} + +#[wasm_bindgen_test] +fn encode_accepts_every_aggregate_hash_policy_name() { + for name in ["auto", "none", "header", "footer", "both"] { + assert!( + encode_with_aggregate_hash(Some(name)).is_ok(), + "policy '{name}' must be accepted" + ); + } + assert!( + encode_with_aggregate_hash(None).is_ok(), + "an omitted policy means auto" + ); +} + +#[wasm_bindgen_test] +fn encode_rejects_an_unknown_aggregate_hash_policy_name() { + // The TypeScript wrapper validates first, so this boundary guard is + // only reachable through the raw WASM export — keep it honest here. + let err = encode_with_aggregate_hash(Some("sideways")).unwrap_err(); + let message = js_sys::Error::from(err).message().as_string().unwrap(); + assert!( + message.contains("unknown aggregateHash policy 'sideways'"), + "unexpected message: {message}" + ); + assert!( + message.contains("auto, none, header, footer, both"), + "the error must list the accepted names: {message}" + ); +} diff --git a/rust/tensogram/src/frame_walk.rs b/rust/tensogram/src/frame_walk.rs new file mode 100644 index 00000000..c307fc88 --- /dev/null +++ b/rust/tensogram/src/frame_walk.rs @@ -0,0 +1,463 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +//! Structural introspection of a message: walk its frames and read its +//! envelope, without decoding payloads or CBOR. +//! +//! This is the reference implementation every binding mirrors (see +//! `plans/INTERFACE_SYMMETRY.md` §9). Two entry points: +//! +//! - [`message_header`] — the 24-byte preamble as typed values: wire +//! [`version`](MessageHeader::version), [`total_length`](MessageHeader::total_length), +//! and the eight structural predicates (`has_header_metadata`, …). Tells you +//! whether a message is random-access (metadata/index/hashes in the *header*) +//! or streaming (in the *footer*) without reading a single frame. +//! - [`frames`] — a lazy iterator over the message's frames (types 1–9). The +//! preamble and postamble are *not* frames and are not yielded; use +//! [`message_header`] for the envelope. +//! +//! # Borrowing +//! +//! [`FrameInfo::payload`] borrows the caller's message buffer — it is a view, +//! not a copy, and is valid for as long as that buffer lives. The iterator +//! itself is just a cursor. + +use crate::error::{Result, TensogramError}; +use crate::wire::{ + END_MAGIC, FRAME_HEADER_SIZE, FrameFlags, FrameHeader, FrameType, MessageFlags, POSTAMBLE_SIZE, + PREAMBLE_SIZE, Preamble, footer_size_for, +}; + +/// One frame's structural description plus a borrowed view of its content. +#[derive(Debug, Clone, Copy)] +pub struct FrameInfo<'a> { + /// The frame's type (1–9). + pub frame_type: FrameType, + /// Frame-type-specific version field from the frame header. + pub version: u16, + /// Raw 16-bit frame flags (bit 1 is `HASH_PRESENT`; see + /// [`has_hash`](Self::has_hash)). + pub flags: u16, + /// Byte offset of the frame header, **relative to the start of the + /// message** that was passed to [`frames`]. + pub offset: usize, + /// Whole-frame span in bytes: frame header through `ENDF`, excluding any + /// alignment padding that follows. + pub length: usize, + /// The frame's content: everything between the 16-byte frame header and + /// the type-specific footer. + /// + /// For a data-object frame (type 9) this is the encoded payload *and* the + /// trailing CBOR descriptor (the 20-byte footer `[cbor_offset][hash][ENDF]` + /// is excluded). For every other frame type it is the CBOR body (the + /// 12-byte footer `[hash][ENDF]` is excluded). Use + /// [`offset`](Self::offset)/[`length`](Self::length) for the whole frame. + pub payload: &'a [u8], +} + +impl FrameInfo<'_> { + /// `true` if this frame's `HASH_PRESENT` flag is set, i.e. its hash slot + /// holds a meaningful digest. + #[must_use] + pub fn has_hash(&self) -> bool { + self.flags & FrameFlags::HASH_PRESENT != 0 + } +} + +/// A message's envelope (preamble), decoded into typed values. +#[derive(Debug, Clone, Copy)] +pub struct MessageHeader { + /// Wire-format version (3 in v3). + pub version: u16, + /// Total message length in bytes, preamble through postamble. `0` in a + /// streaming message whose length was never back-filled. + pub total_length: u64, + /// Raw preamble flag bits; prefer the `has_*` predicates. + pub flags: MessageFlags, +} + +impl MessageHeader { + /// A `HeaderMetadata` frame is present (random-access mode). + #[must_use] + pub fn has_header_metadata(&self) -> bool { + self.flags.has(MessageFlags::HEADER_METADATA) + } + /// A `FooterMetadata` frame is present (streaming mode). + #[must_use] + pub fn has_footer_metadata(&self) -> bool { + self.flags.has(MessageFlags::FOOTER_METADATA) + } + /// A `HeaderIndex` frame is present. + #[must_use] + pub fn has_header_index(&self) -> bool { + self.flags.has(MessageFlags::HEADER_INDEX) + } + /// A `FooterIndex` frame is present. + #[must_use] + pub fn has_footer_index(&self) -> bool { + self.flags.has(MessageFlags::FOOTER_INDEX) + } + /// A `HeaderHash` frame is present. + #[must_use] + pub fn has_header_hashes(&self) -> bool { + self.flags.has(MessageFlags::HEADER_HASHES) + } + /// A `FooterHash` frame is present. + #[must_use] + pub fn has_footer_hashes(&self) -> bool { + self.flags.has(MessageFlags::FOOTER_HASHES) + } + /// At least one `PrecederMetadata` frame appears in the body. + #[must_use] + pub fn has_preceder_metadata(&self) -> bool { + self.flags.has(MessageFlags::PRECEDER_METADATA) + } + /// Advisory: every frame in this message has its per-frame `HASH_PRESENT` + /// bit set. The per-frame bit remains authoritative for any single frame. + #[must_use] + pub fn has_hashes_present(&self) -> bool { + self.flags.has(MessageFlags::HASHES_PRESENT) + } +} + +/// Read a message's envelope without walking its frames. +/// +/// `message` must start at the `TENSOGRM` preamble magic. +pub fn message_header(message: &[u8]) -> Result { + let p = Preamble::read_from(message)?; + Ok(MessageHeader { + version: p.version, + total_length: p.total_length, + flags: p.flags, + }) +} + +/// Lazily iterate the frames of one message. +/// +/// `message` must start at the `TENSOGRM` preamble magic — typically a slice +/// obtained from [`crate::scan`]. Yields `Err` once and then stops if the frame +/// chain is malformed. +pub fn frames(message: &[u8]) -> Result> { + let p = Preamble::read_from(message)?; + // Bound the walk to the frame region so the postamble is never mistaken + // for a frame. A streaming message that was never back-filled reports + // total_length == 0; fall back to the buffer end. Clamp to the buffer so a + // truncated message surfaces a framing error rather than reading past it. + let end = if p.total_length > 0 { + usize::try_from(p.total_length) + .unwrap_or(usize::MAX) + .saturating_sub(POSTAMBLE_SIZE) + .min(message.len()) + } else if message.len() >= POSTAMBLE_SIZE && message.ends_with(END_MAGIC) { + // Streaming message whose total_length was never back-filled: the + // postamble is still the trailing POSTAMBLE_SIZE bytes (its END_MAGIC + // sits at the very end), so exclude it rather than scanning into it. + message.len() - POSTAMBLE_SIZE + } else { + message.len() + }; + Ok(FrameIter { + buf: message, + pos: PREAMBLE_SIZE, + end, + done: false, + }) +} + +/// Lazy cursor over a message's frames. See [`frames`]. +#[derive(Debug)] +pub struct FrameIter<'a> { + buf: &'a [u8], + pos: usize, + end: usize, + done: bool, +} + +impl<'a> Iterator for FrameIter<'a> { + type Item = Result>; + + fn next(&mut self) -> Option { + if self.done { + return None; + } + // Skip inter-frame alignment padding (zero bytes after ENDF). + while self.pos + 2 <= self.end && &self.buf[self.pos..self.pos + 2] != b"FR" { + self.pos += 1; + } + if self.pos + 2 > self.end { + // Only padding left — a clean end of the frame region. + self.done = true; + return None; + } + // A frame starts here, but the region ends before its header does. + if self.pos + FRAME_HEADER_SIZE > self.end { + self.done = true; + return Some(Err(TensogramError::Framing(format!( + "truncated frame header at offset {}: {} bytes available, {FRAME_HEADER_SIZE} required", + self.pos, + self.end - self.pos + )))); + } + let fh = match FrameHeader::read_from(&self.buf[self.pos..]) { + Ok(fh) => fh, + Err(e) => { + self.done = true; + return Some(Err(e)); + } + }; + let footer = footer_size_for(fh.frame_type); + let min_total = FRAME_HEADER_SIZE + footer; + let total = usize::try_from(fh.total_length).unwrap_or(usize::MAX); + if total < min_total || self.pos + total > self.end { + self.done = true; + return Some(Err(TensogramError::Framing(format!( + "frame at offset {} declares total_length {} which does not fit the message", + self.pos, fh.total_length + )))); + } + let info = FrameInfo { + frame_type: fh.frame_type, + version: fh.version, + flags: fh.flags, + offset: self.pos, + length: total, + payload: &self.buf[self.pos + FRAME_HEADER_SIZE..self.pos + total - footer], + }; + self.pos += total; + Some(Ok(info)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::{ + ByteOrder, DataObjectDescriptor, Dtype, EncodeOptions, GlobalMetadata, StreamingEncoder, + encode, + }; + use std::collections::BTreeMap; + + fn desc(shape: Vec) -> DataObjectDescriptor { + let strides = { + let mut s = vec![1u64; shape.len()]; + for i in (0..shape.len().saturating_sub(1)).rev() { + s[i] = s[i + 1] * shape[i + 1]; + } + s + }; + DataObjectDescriptor { + obj_type: "ntensor".to_string(), + ndim: shape.len() as u64, + shape, + strides, + dtype: Dtype::Float32, + byte_order: ByteOrder::Little, + encoding: "none".to_string(), + filter: "none".to_string(), + compression: "none".to_string(), + params: BTreeMap::new(), + masks: None, + } + } + + /// Buffered encode → random-access layout (metadata/index in the header). + fn buffered_message(n_objects: usize) -> Vec { + let meta = GlobalMetadata::default(); + let d = desc(vec![4]); + let data = vec![0u8; 16]; + let objs: Vec<(&DataObjectDescriptor, &[u8])> = + (0..n_objects).map(|_| (&d, data.as_slice())).collect(); + encode(&meta, &objs, &EncodeOptions::default()).expect("encode") + } + + /// Streaming encode → footer-side index/hashes. + fn streamed_message() -> Vec { + let meta = GlobalMetadata::default(); + let d = desc(vec![4]); + let data = vec![0u8; 16]; + let mut enc = StreamingEncoder::new( + std::io::Cursor::new(Vec::new()), + &meta, + &EncodeOptions::default(), + ) + .expect("streaming new"); + enc.write_object(&d, &data).expect("write"); + enc.finish().expect("finish").into_inner() + } + + fn collect(msg: &[u8]) -> Vec> { + frames(msg) + .expect("frames") + .map(|r| r.expect("frame ok")) + .collect() + } + + #[test] + fn walks_frames_and_finds_one_data_object_per_object() { + let msg = buffered_message(3); + let fs = collect(&msg); + assert!(!fs.is_empty(), "expected frames"); + let data_objects = fs.iter().filter(|f| f.frame_type.is_data_object()).count(); + assert_eq!(data_objects, 3, "one data-object frame per encoded object"); + // The preamble/postamble are NOT frames. + assert!( + fs[0].offset >= PREAMBLE_SIZE, + "first frame starts after the preamble" + ); + } + + #[test] + fn frames_are_ordered_in_bounds_and_do_not_overlap() { + let msg = buffered_message(2); + let fs = collect(&msg); + let mut prev_end = PREAMBLE_SIZE; + for f in &fs { + assert!(f.offset >= prev_end, "frames must not overlap"); + assert!(f.offset + f.length <= msg.len(), "frame must be in bounds"); + assert!( + f.length >= FRAME_HEADER_SIZE, + "frame spans at least its header" + ); + prev_end = f.offset + f.length; + } + } + + #[test] + fn payload_excludes_header_and_footer_and_lies_inside_the_frame() { + let msg = buffered_message(1); + for f in collect(&msg) { + // Content starts after the 16-byte frame header ... + let frame = &msg[f.offset..f.offset + f.length]; + let start = FRAME_HEADER_SIZE; + let end = start + f.payload.len(); + assert!(end <= frame.len(), "payload must fit inside the frame"); + assert_eq!( + f.payload, + &frame[start..end], + "payload is the content slice" + ); + // ... and stops before the footer, which always ends with ENDF. + assert_eq!(&frame[frame.len() - 4..], b"ENDF"); + assert!( + f.payload.len() < f.length - FRAME_HEADER_SIZE, + "payload must exclude the type-specific footer" + ); + } + } + + #[test] + fn buffered_message_header_flags_match_its_frames_exactly() { + // In buffered (random-access) mode the encoder knows the whole message + // up front, so every preamble flag is an exact statement about the + // frames present. + let msg = buffered_message(2); + let h = message_header(&msg).expect("header"); + assert_eq!(h.version, crate::WIRE_VERSION); + assert_eq!(h.total_length as usize, msg.len()); + let types: Vec = collect(&msg).iter().map(|f| f.frame_type).collect(); + let has = |t: FrameType| types.contains(&t); + + assert_eq!(h.has_header_metadata(), has(FrameType::HeaderMetadata)); + assert_eq!(h.has_footer_metadata(), has(FrameType::FooterMetadata)); + assert_eq!(h.has_header_index(), has(FrameType::HeaderIndex)); + assert_eq!(h.has_footer_index(), has(FrameType::FooterIndex)); + assert_eq!(h.has_header_hashes(), has(FrameType::HeaderHash)); + assert_eq!(h.has_footer_hashes(), has(FrameType::FooterHash)); + assert_eq!(h.has_preceder_metadata(), has(FrameType::PrecederMetadata)); + } + + #[test] + fn streaming_message_header_flags_are_advisory_but_never_understate() { + // In streaming mode the preamble is written before any object, so the + // encoder cannot know what will follow: it sets PRECEDER_METADATA + // optimistically (see `build_preamble_and_header_bytes`) and may leave + // total_length at 0. The invariant that still holds in BOTH directions + // for the frames it can predict, and one-directionally for the rest, is: + // a frame that IS present must be advertised. + let msg = streamed_message(); + let h = message_header(&msg).expect("header"); + assert_eq!(h.version, crate::WIRE_VERSION); + let types: Vec = collect(&msg).iter().map(|f| f.frame_type).collect(); + let implies = |present: bool, flag: bool| !present || flag; + + assert!(implies( + types.contains(&FrameType::HeaderMetadata), + h.has_header_metadata() + )); + assert!(implies( + types.contains(&FrameType::FooterMetadata), + h.has_footer_metadata() + )); + assert!(implies( + types.contains(&FrameType::HeaderIndex), + h.has_header_index() + )); + assert!(implies( + types.contains(&FrameType::FooterIndex), + h.has_footer_index() + )); + assert!(implies( + types.contains(&FrameType::HeaderHash), + h.has_header_hashes() + )); + assert!(implies( + types.contains(&FrameType::FooterHash), + h.has_footer_hashes() + )); + assert!(implies( + types.contains(&FrameType::PrecederMetadata), + h.has_preceder_metadata() + )); + // The walk must still stop cleanly at the postamble even though + // total_length was never back-filled. + assert!( + types.contains(&FrameType::FooterIndex), + "footer frames are reachable" + ); + } + + #[test] + fn has_hash_matches_the_per_frame_flag() { + // Default options hash frames, so at least one frame must advertise it, + // and every frame's has_hash() must mirror its raw flag bit. + let msg = buffered_message(1); + let fs = collect(&msg); + assert!( + fs.iter().any(FrameInfo::has_hash), + "default encode hashes frames" + ); + for f in &fs { + assert_eq!(f.has_hash(), f.flags & (1 << 1) != 0); + } + } + + #[test] + fn rejects_a_buffer_that_is_not_a_message() { + assert!(message_header(b"not a tensogram message").is_err()); + assert!(frames(b"not a tensogram message").is_err()); + assert!(message_header(&[]).is_err()); + } + + #[test] + fn truncated_frame_chain_yields_an_error_item_then_stops() { + let msg = buffered_message(2); + // Cut *inside* the last frame: the preamble stays valid, the chain does not. + let last = *collect(&msg).last().expect("at least one frame"); + let truncated = &msg[..last.offset + 8]; + let mut it = frames(truncated).expect("preamble still parses"); + let mut saw_err = false; + for item in it.by_ref() { + if item.is_err() { + saw_err = true; + break; + } + } + assert!(saw_err, "a truncated chain must surface an error"); + assert!(it.next().is_none(), "iteration stops after an error"); + } +} diff --git a/rust/tensogram/src/lib.rs b/rust/tensogram/src/lib.rs index 5fbd3f89..e1f7048a 100644 --- a/rust/tensogram/src/lib.rs +++ b/rust/tensogram/src/lib.rs @@ -12,6 +12,7 @@ pub mod dtype; pub mod encode; pub mod error; pub mod file; +pub mod frame_walk; pub mod framing; pub mod hash; pub mod iter; @@ -45,6 +46,7 @@ pub use dtype::Dtype; pub use encode::{AggregateHashPolicy, EncodeOptions, encode, encode_pre_encoded}; pub use error::{Result, TensogramError}; pub use file::{MessageLayout, TensogramFile}; +pub use frame_walk::{FrameInfo, FrameIter, MessageHeader, frames, message_header}; pub use framing::{ ScanOptions, data_object_inline_hashes, scan, scan_file, scan_file_with_options, scan_with_options, diff --git a/typescript/src/encode.ts b/typescript/src/encode.ts index 8af566cd..dd0b7c95 100644 --- a/typescript/src/encode.ts +++ b/typescript/src/encode.ts @@ -9,9 +9,16 @@ /** * Encode wrapper. * - * Translates the ergonomic TS surface into the shape the wasm-bindgen - * `encode()` export expects: - * `(metadata: any, objects: Array<{descriptor, data}>, hash?: boolean)` + * Translates the ergonomic TS surface into the positional argument list + * the wasm-bindgen `encode()` export expects: metadata, the + * `{descriptor, data}` array, then the option scalars (hash, the + * NaN / Inf mask knobs, aggregate-hash placement). + * + * Option names that the WASM side does not parse itself — + * `aggregateHash` and the platform-no-op `compressionBackend` — are + * validated here so a typo surfaces as an {@link InvalidArgumentError} + * naming the offending option rather than as a generic framing error + * from across the boundary. */ import { rethrowTyped, InvalidArgumentError } from './errors.js'; @@ -21,7 +28,36 @@ import { assertValidMetadata, toUint8View, } from './internal/validation.js'; -import type { EncodeInput, EncodeOptions, GlobalMetadata } from './types.js'; +import type { + AggregateHashPolicy, + CompressionBackend, + EncodeInput, + EncodeOptions, + GlobalMetadata, +} from './types.js'; + +/** + * Accepted {@link AggregateHashPolicy} names, for runtime validation of + * plain-JS callers who bypass the compile-time union. + * + * Declared as a `Record` keyed by the union so the compiler rejects + * this table the moment a policy is added to — or removed from — the + * type: the runtime mirror cannot drift from `AggregateHashPolicy`. + */ +const AGGREGATE_HASH_POLICIES: Record = { + auto: true, + none: true, + header: true, + footer: true, + both: true, +}; + +/** Accepted {@link CompressionBackend} names. See above for the shape. */ +const COMPRESSION_BACKENDS: Record = { + auto: true, + ffi: true, + pure: true, +}; /** * Encode global metadata + a list of `(descriptor, data)` pairs into a @@ -30,12 +66,15 @@ import type { EncodeInput, EncodeOptions, GlobalMetadata } from './types.js'; * @param metadata - Global metadata (free-form CBOR; only `base`, `_reserved_`, and `_extra_` are library-interpreted) * @param objects - Data objects; each `data` is any `ArrayBufferView` * (`TypedArray`, `DataView`, ...) in native byte order - * @param options - Optional hash selection and strict-finite flags + * @param options - Optional hash selection, strict-finite flags, and + * aggregate-hash placement. See {@link EncodeOptions}. * @returns Wire-format bytes as a `Uint8Array` * @throws {MetadataError} if metadata is malformed * @throws {EncodingError} if a pipeline stage rejects the input (e.g. NaN in simple_packing) * or if the strict-finite check catches a NaN/Inf * @throws {CompressionError} if a compression codec fails + * @throws {InvalidArgumentError} if `aggregateHash` or `compressionBackend` + * names an unknown value */ export function encode( metadata: GlobalMetadata, @@ -44,6 +83,20 @@ export function encode( ): Uint8Array { assertValidMetadata(metadata); assertValidObjects(objects); + assertKnownName( + options?.aggregateHash, + AGGREGATE_HASH_POLICIES, + 'aggregateHash', + ); + // Validated but deliberately NOT forwarded: the WASM bundle carries + // pure-Rust codecs only, so no value can change the bytes produced. + // See `EncodeOptions.compressionBackend` for the platform-no-op + // rationale — we still reject typos rather than swallow them. + assertKnownName( + options?.compressionBackend, + COMPRESSION_BACKENDS, + 'compressionBackend', + ); const wbg = getWbg(); const objArray = objects.map((o) => ({ @@ -64,10 +117,29 @@ export function encode( options?.posInfMaskMethod, options?.negInfMaskMethod, options?.smallMaskThresholdBytes, + options?.aggregateHash, ), ); } +/** + * Reject an option value that is not in its accepted-name table. An + * `undefined` value means "not supplied" and always passes — the + * encoder applies its own default. + */ +function assertKnownName( + value: string | undefined, + accepted: Record, + option: string, +): void { + if (value === undefined) return; + if (!Object.hasOwn(accepted, value)) { + throw new InvalidArgumentError( + `unknown ${option} '${value}', expected one of: ${Object.keys(accepted).join(', ')}`, + ); + } +} + function assertValidObjects(objects: readonly EncodeInput[]): void { if (!Array.isArray(objects)) { throw new InvalidArgumentError(`objects must be an array, got ${typeof objects}`); diff --git a/typescript/src/file.ts b/typescript/src/file.ts index 9d7128c6..e65919ad 100644 --- a/typescript/src/file.ts +++ b/typescript/src/file.ts @@ -1192,6 +1192,8 @@ export class TensogramFile implements AsyncIterable { posInfMaskMethod: options.posInfMaskMethod, negInfMaskMethod: options.negInfMaskMethod, smallMaskThresholdBytes: options.smallMaskThresholdBytes, + aggregateHash: options.aggregateHash, + compressionBackend: options.compressionBackend, }; const bytes = rethrowTyped(() => encode(metadata, objects, encodeOpts)); diff --git a/typescript/src/frameWalk.ts b/typescript/src/frameWalk.ts new file mode 100644 index 00000000..290eac5f --- /dev/null +++ b/typescript/src/frameWalk.ts @@ -0,0 +1,156 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// In applying this licence, ECMWF does not waive the privileges and immunities +// granted to it by virtue of its status as an intergovernmental organisation nor +// does it submit to any jurisdiction. + +/** + * Structural introspection of a message: walk its frames and read its + * envelope, without decoding payloads or CBOR. + * + * Mirror the Rust core `tensogram::frames` / `tensogram::message_header` + * (`rust/tensogram/src/frame_walk.rs`): + * + * - {@link messageHeader} reads the 24-byte preamble as typed values — + * wire version, total length, and eight structural predicates. Tells + * you whether a message is random-access (metadata / index / hashes + * in the *header*) or streaming (in the *footer*) without touching a + * single frame. + * - {@link frames} walks the message's frames (types 1–9). The + * preamble and postamble are **not** frames and are never returned. + * + * ## One message per call + * + * Both functions take the bytes of a **single** message, starting at + * its `TENSOGRM` preamble magic. A `.tgm` file is a concatenation of + * messages, so callers holding a whole file must locate the boundaries + * with {@link scan} and slice first: + * + * ```typescript + * for (const { offset, length } of scan(fileBytes)) { + * const message = fileBytes.subarray(offset, offset + length); + * for (const frame of frames(message)) { + * // frame.offset is relative to `message`, not to `fileBytes` + * console.log(frame.frameType, offset + frame.offset, frame.length); + * } + * } + * ``` + */ + +import { getWbg } from './init.js'; +import { InvalidArgumentError, rethrowTyped } from './errors.js'; +import { safeNumberFromBigint } from './internal/layout.js'; +import type { Frame, FrameTypeName, MessageHeader } from './types.js'; + +/** Shape of one entry in the wasm `frames()` array. */ +interface WbgFrame { + frame_type: FrameTypeName; + frame_type_code: number; + version: number; + flags: number; + offset: number; + length: number; + payload: Uint8Array; + has_hash: boolean; +} + +/** Shape of the wasm `message_header()` result. */ +interface WbgMessageHeader { + version: number; + total_length: number | bigint; + has_header_metadata: boolean; + has_footer_metadata: boolean; + has_header_index: boolean; + has_footer_index: boolean; + has_header_hashes: boolean; + has_footer_hashes: boolean; + has_preceder_metadata: boolean; + has_hashes_present: boolean; +} + +/** + * Walk the frames of a **single** Tensogram message, in wire order. + * + * Returns an eagerly-materialised array rather than a generator: a + * message holds only a handful of frames (the header frames plus one + * per data object), and the underlying WASM call already materialises + * the whole walk. A lazily-pulled iterator would promise a laziness + * this binding cannot deliver. The consequence is that a malformed + * frame chain **throws** instead of yielding a short array — corruption + * is never silently rendered as "fewer frames". + * + * Each {@link Frame.payload} is a copy on the JS heap, not a view into + * WASM linear memory, so it stays valid across later WASM calls and can + * be mutated freely (see {@link Frame.payload} for the full contract). + * + * @param buf - Wire-format bytes of ONE message, starting at the + * `TENSOGRM` preamble magic. Use {@link scan} on a multi-message + * buffer and slice before calling; frame offsets are relative to + * `buf`. + * @returns One {@link Frame} per frame, in wire order. The preamble + * and postamble are not frames and are not included. + * @throws {InvalidArgumentError} if `buf` is not a `Uint8Array`. + * @throws {FramingError} if `buf` does not start with a valid preamble, + * or if the frame chain is malformed (truncated frame header, a + * declared frame length that does not fit the message). + */ +export function frames(buf: Uint8Array): Frame[] { + assertUint8Array(buf, 'buf'); + const wbg = getWbg(); + const raw = rethrowTyped(() => wbg.frames(buf) as unknown as WbgFrame[]); + return raw.map((f) => ({ + frameType: f.frame_type, + frameTypeCode: f.frame_type_code, + version: f.version, + flags: f.flags, + offset: f.offset, + length: f.length, + payload: f.payload, + hasHash: f.has_hash, + })); +} + +/** + * Read a **single** message's envelope without walking its frames. + * + * The eight `has*` predicates are exact for a buffered message (one + * produced by {@link encode}): the encoder knows the whole message up + * front, so each flag is a precise statement about the frames present. + * A {@link StreamingEncoder} writes the preamble before any object, so + * there the flags are advisory — only `frame present ⇒ flag set` holds + * (the encoder sets `PRECEDER_METADATA` optimistically, and + * {@link MessageHeader.totalLength} may stay `0` when the sink was not + * seekable at `finish()` time). + * + * @param buf - Wire-format bytes of ONE message, starting at the + * `TENSOGRM` preamble magic. Use {@link scan} on a multi-message + * buffer and slice before calling. + * @throws {InvalidArgumentError} if `buf` is not a `Uint8Array`, or if + * the message advertises a length above `Number.MAX_SAFE_INTEGER`. + * @throws {FramingError} if `buf` does not start with a valid preamble. + */ +export function messageHeader(buf: Uint8Array): MessageHeader { + assertUint8Array(buf, 'buf'); + const wbg = getWbg(); + const h = rethrowTyped(() => wbg.message_header(buf) as unknown as WbgMessageHeader); + return { + version: h.version, + totalLength: safeNumberFromBigint(h.total_length, 'messageHeader.totalLength'), + hasHeaderMetadata: h.has_header_metadata, + hasFooterMetadata: h.has_footer_metadata, + hasHeaderIndex: h.has_header_index, + hasFooterIndex: h.has_footer_index, + hasHeaderHashes: h.has_header_hashes, + hasFooterHashes: h.has_footer_hashes, + hasPrecederMetadata: h.has_preceder_metadata, + hasHashesPresent: h.has_hashes_present, + }; +} + +function assertUint8Array(buf: unknown, name: string): asserts buf is Uint8Array { + if (!(buf instanceof Uint8Array)) { + throw new InvalidArgumentError(`${name} must be a Uint8Array, got ${typeof buf}`); + } +} diff --git a/typescript/src/index.ts b/typescript/src/index.ts index 78361b33..18aa4afa 100644 --- a/typescript/src/index.ts +++ b/typescript/src/index.ts @@ -18,6 +18,8 @@ * {@link scanWithOptions} — whole-buffer decoding with dtype-aware payload views * - {@link decodeWithMasks} — advanced decode returning raw NaN / Inf masks * alongside the `0.0`-substituted payloads + * - {@link frames}, {@link messageHeader} — structural introspection of ONE + * message: its frame chain and its typed envelope, without decoding * - {@link objects}, {@link objectsMetadata} — lazy per-object iterators over * one message (decode on demand / descriptors without payload decode) * - {@link dataObjectInlineHashes} — read per-object inline integrity digests @@ -51,6 +53,7 @@ export const WIRE_VERSION = 3; export { encode } from './encode.js'; export { decode, decodeMetadata, decodeObject, scan, scanWithOptions } from './decode.js'; export { decodeWithMasks } from './decodeMasks.js'; +export { frames, messageHeader } from './frameWalk.js'; export { objects, objectsMetadata } from './iter.js'; export { decodeStream } from './streaming.js'; export { TensogramFile } from './file.js'; @@ -136,11 +139,13 @@ export { } from './errors.js'; export type { + AggregateHashPolicy, AppendOptions, BaseEntry, ByteOrder, CborValue, Compression, + CompressionBackend, DataObjectDescriptor, DecodedFrame, DecodedMaskSet, @@ -161,12 +166,15 @@ export type { FileSource, FileValidationReport, Filter, + Frame, + FrameTypeName, FromUrlOptions, GlobalMetadata, HashDescriptor, IssueCode, IssueSeverity, MaskMethod, + MessageHeader, MessagePosition, OpenFileOptions, PreEncodedInput, diff --git a/typescript/src/types.ts b/typescript/src/types.ts index 75beae6f..2f2a3ad2 100644 --- a/typescript/src/types.ts +++ b/typescript/src/types.ts @@ -383,6 +383,108 @@ export interface MessagePosition { length: number; } +/** + * Canonical wire-format name of a frame type, as spelled in + * `plans/WIRE_FORMAT.md` §6 and in the Rust `FrameType` enum. + * + * Type 4 is reserved (it held the obsolete v2 tensor frame) and never + * appears — a v3 reader that meets one raises a {@link FramingError}. + */ +export type FrameTypeName = + | 'HeaderMetadata' + | 'HeaderIndex' + | 'HeaderHash' + | 'FooterHash' + | 'FooterIndex' + | 'FooterMetadata' + | 'PrecederMetadata' + | 'NTensorFrame'; + +/** + * One frame of a message, as returned by {@link frames}. + * + * Mirrors the Rust core `tensogram::FrameInfo`. + */ +export interface Frame { + /** Canonical frame-type name. See {@link FrameTypeName}. */ + frameType: FrameTypeName; + /** The frame type's wire number (1–3, 5–9). */ + frameTypeCode: number; + /** Frame-type-specific version field from the frame header. */ + version: number; + /** + * Raw 16-bit frame flags. Bit 1 is `HASH_PRESENT`, surfaced as + * {@link Frame.hasHash}. + */ + flags: number; + /** + * Byte offset of the frame header, **relative to the start of the + * message** passed to {@link frames} — not to a multi-message file. + */ + offset: number; + /** + * Whole-frame span in bytes: the frame header through its `ENDF` + * magic, excluding any inter-frame alignment padding that follows. + */ + length: number; + /** + * The frame's content: everything between the 16-byte frame header + * and the type-specific footer (20 bytes for `NTensorFrame`, 12 for + * every other type). For an `NTensorFrame` this is the encoded + * payload plus any mask blobs plus the trailing CBOR descriptor; for + * every other frame type it is the CBOR body. + * + * Always a **copy on the JS heap**, never a view into WASM linear + * memory: safe to retain across later WASM calls (which may grow that + * memory) and safe to mutate without touching the caller's buffer. + * Same ownership contract as the mask arrays from + * {@link decodeWithMasks}. Use {@link Frame.offset} / + * {@link Frame.length} when you want the whole frame instead. + */ + payload: Uint8Array; + /** + * `true` when this frame's `HASH_PRESENT` flag is set, i.e. its hash + * slot holds a meaningful digest. + */ + hasHash: boolean; +} + +/** + * A message's envelope (its 24-byte preamble) decoded into typed + * values, as returned by {@link messageHeader}. + * + * Mirrors the Rust core `tensogram::MessageHeader`. + */ +export interface MessageHeader { + /** Wire-format version — always {@link WIRE_VERSION} in v3. */ + version: number; + /** + * Total message length in bytes, preamble through postamble. `0` for + * a streaming message whose length was never back-filled. + */ + totalLength: number; + /** A `HeaderMetadata` frame is present (random-access mode). */ + hasHeaderMetadata: boolean; + /** A `FooterMetadata` frame is present (streaming mode). */ + hasFooterMetadata: boolean; + /** A `HeaderIndex` frame is present. */ + hasHeaderIndex: boolean; + /** A `FooterIndex` frame is present. */ + hasFooterIndex: boolean; + /** A `HeaderHash` frame is present. */ + hasHeaderHashes: boolean; + /** A `FooterHash` frame is present. */ + hasFooterHashes: boolean; + /** At least one `PrecederMetadata` frame appears in the body. */ + hasPrecederMetadata: boolean; + /** + * Advisory: every frame in this message has its per-frame + * `HASH_PRESENT` bit set. {@link Frame.hasHash} remains + * authoritative for any single frame. + */ + hasHashesPresent: boolean; +} + /** * Options for {@link scanWithOptions}. * @@ -419,6 +521,51 @@ export type MaskMethod = | 'zstd' | 'blosc2'; +/** + * Where the aggregate hash frame is written inside a message. + * + * The aggregate frame is a CBOR list of every per-object inline hash + * slot — a redundant copy that lets tools read all digests at once + * without walking the body. The per-frame inline slots always remain + * authoritative (see {@link dataObjectInlineHashes}). + * + * Mirrors the Rust core `AggregateHashPolicy`: + * + * - `"auto"` — the encoder picks; buffered {@link encode} writes a + * `HeaderHash` frame. This is the default. + * - `"none"` — no aggregate frame at all. + * - `"header"` — a `HeaderHash` frame only. + * - `"footer"` — a `FooterHash` frame only. + * - `"both"` — a `HeaderHash` **and** a `FooterHash` frame, carrying + * identical hash lists. + * + * Ignored when `hash: false`: with per-frame hashing disabled there is + * nothing to aggregate, so no frame is written whichever policy is + * requested. Use {@link frames} to observe the resulting placement. + */ +export type AggregateHashPolicy = 'auto' | 'none' | 'header' | 'footer' | 'both'; + +/** + * Which compression implementation the encode pipeline should use. + * + * Mirrors the Rust core `CompressionBackend` (`"auto"` consults the + * `TENSOGRAM_COMPRESSION_BACKEND` environment variable and the platform + * default; `"ffi"` selects the C libraries — libaec, libzstd; `"pure"` + * selects the pure-Rust codecs). + * + * **Platform no-op in these bindings.** The WASM bundle is compiled + * with the pure-Rust codecs only (`lz4`, `szip-pure`, `zstd-pure`); + * there is no C library to link against in a browser or in Node's WASM + * sandbox. Every accepted value therefore produces byte-identical + * output here. The knob exists for source symmetry with the Rust, + * Python, and C bindings — code written against one binding reads the + * same against another — and is validated (an unrecognised name + * throws) but never changes behaviour. The same accepted asymmetry + * applies to the core's `threads` knob, which the single-threaded WASM + * build does not expose at all. + */ +export type CompressionBackend = 'auto' | 'ffi' | 'pure'; + /** Options for `encode()`. */ export interface EncodeOptions { /** @@ -452,6 +599,19 @@ export interface EncodeOptions { * Default `128`. Set to `0` to disable the auto-fallback. */ smallMaskThresholdBytes?: number; + /** + * Where to emit the aggregate hash frame. Default `"auto"`, which + * this buffered encoder resolves to a `HeaderHash` frame. See + * {@link AggregateHashPolicy}. + */ + aggregateHash?: AggregateHashPolicy; + /** + * Which compression implementation to use. **No-op in the WASM + * bindings** — this build carries pure-Rust codecs only. Accepted + * and validated for source symmetry with the other bindings; see + * {@link CompressionBackend}. + */ + compressionBackend?: CompressionBackend; } /** Options for `decode()` / `decodeObject()`. */ @@ -774,6 +934,10 @@ export interface AppendOptions { negInfMaskMethod?: MaskMethod; /** See {@link EncodeOptions.smallMaskThresholdBytes}. Default `128`. */ smallMaskThresholdBytes?: number; + /** See {@link EncodeOptions.aggregateHash}. Default `"auto"`. */ + aggregateHash?: AggregateHashPolicy; + /** See {@link EncodeOptions.compressionBackend}. No-op on WASM. */ + compressionBackend?: CompressionBackend; } /** Options for {@link StreamingEncoder}. */ diff --git a/typescript/tests/encodeAggregateHash.test.ts b/typescript/tests/encodeAggregateHash.test.ts new file mode 100644 index 00000000..482d505d --- /dev/null +++ b/typescript/tests/encodeAggregateHash.test.ts @@ -0,0 +1,250 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// +// Tests for the `aggregateHash` and `compressionBackend` encode knobs. +// Mirror the Rust core `EncodeOptions { aggregate_hash, compression_backend }` +// (`rust/tensogram/src/encode.rs`). The closed loop is verified through the +// frame walker: the requested policy must be observable on the wire. + +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + InvalidArgumentError, + TensogramFile, + decode, + encode, + frames, + init, + messageHeader, +} from '../src/index.js'; +import type { + AggregateHashPolicy, + CompressionBackend, + EncodeOptions, + FrameTypeName, +} from '../src/index.js'; +import { defaultMeta, makeDescriptor } from './helpers.js'; + +function message(opts?: EncodeOptions): Uint8Array { + return encode( + defaultMeta(), + [ + { descriptor: makeDescriptor([4], 'float32'), data: new Float32Array([1, 2, 3, 4]) }, + { descriptor: makeDescriptor([4], 'float32'), data: new Float32Array([5, 6, 7, 8]) }, + ], + opts, + ); +} + +function frameTypes(msg: Uint8Array): FrameTypeName[] { + return frames(msg).map((f) => f.frameType); +} + +describe('EncodeOptions.aggregateHash', () => { + it('defaults to the buffered-mode header placement', async () => { + await init(); + const types = frameTypes(message()); + expect(types.filter((t) => t === 'HeaderHash')).toHaveLength(1); + expect(types).not.toContain('FooterHash'); + }); + + it("'auto' matches the default", async () => { + await init(); + expect(frameTypes(message({ aggregateHash: 'auto' }))).toEqual(frameTypes(message())); + }); + + it("'none' emits no aggregate hash frame at all", async () => { + await init(); + const msg = message({ aggregateHash: 'none' }); + const types = frameTypes(msg); + expect(types).not.toContain('HeaderHash'); + expect(types).not.toContain('FooterHash'); + const h = messageHeader(msg); + expect(h.hasHeaderHashes).toBe(false); + expect(h.hasFooterHashes).toBe(false); + // Per-frame inline hashes are unaffected. + expect(h.hasHashesPresent).toBe(true); + }); + + it("'header' emits a HeaderHash frame only", async () => { + await init(); + const msg = message({ aggregateHash: 'header' }); + const types = frameTypes(msg); + expect(types.filter((t) => t === 'HeaderHash')).toHaveLength(1); + expect(types).not.toContain('FooterHash'); + const h = messageHeader(msg); + expect(h.hasHeaderHashes).toBe(true); + expect(h.hasFooterHashes).toBe(false); + }); + + it("'footer' emits a FooterHash frame only", async () => { + await init(); + const msg = message({ aggregateHash: 'footer' }); + const types = frameTypes(msg); + expect(types).not.toContain('HeaderHash'); + expect(types.filter((t) => t === 'FooterHash')).toHaveLength(1); + const h = messageHeader(msg); + expect(h.hasHeaderHashes).toBe(false); + expect(h.hasFooterHashes).toBe(true); + }); + + it("'both' puts a hash frame in BOTH the header and the footer", async () => { + await init(); + const msg = message({ aggregateHash: 'both' }); + const types = frameTypes(msg); + expect(types.filter((t) => t === 'HeaderHash')).toHaveLength(1); + expect(types.filter((t) => t === 'FooterHash')).toHaveLength(1); + // The header aggregate precedes the data objects; the footer + // aggregate follows them. + expect(types.indexOf('HeaderHash')).toBeLessThan(types.indexOf('NTensorFrame')); + expect(types.indexOf('FooterHash')).toBeGreaterThan(types.lastIndexOf('NTensorFrame')); + const h = messageHeader(msg); + expect(h.hasHeaderHashes).toBe(true); + expect(h.hasFooterHashes).toBe(true); + }); + + it('carries identical aggregate payloads in header and footer', async () => { + await init(); + const fs = frames(message({ aggregateHash: 'both' })); + const header = fs.find((f) => f.frameType === 'HeaderHash'); + const footer = fs.find((f) => f.frameType === 'FooterHash'); + expect(header).toBeDefined(); + expect(footer).toBeDefined(); + expect(Array.from(header!.payload)).toEqual(Array.from(footer!.payload)); + }); + + it('round-trips through decode under every policy', async () => { + await init(); + const policies: AggregateHashPolicy[] = ['auto', 'none', 'header', 'footer', 'both']; + for (const aggregateHash of policies) { + const msg = decode(message({ aggregateHash })); + try { + expect(msg.objects).toHaveLength(2); + expect(Array.from(msg.objects[0].data() as Float32Array)).toEqual([1, 2, 3, 4]); + expect(Array.from(msg.objects[1].data() as Float32Array)).toEqual([5, 6, 7, 8]); + } finally { + msg.close(); + } + } + }); + + it('is ignored when hashing is disabled (nothing to aggregate)', async () => { + await init(); + const types = frameTypes(message({ hash: false, aggregateHash: 'both' })); + expect(types).not.toContain('HeaderHash'); + expect(types).not.toContain('FooterHash'); + }); + + it('rejects an unknown policy name', async () => { + await init(); + // @ts-expect-error deliberate invalid policy + expect(() => message({ aggregateHash: 'sideways' })).toThrow(InvalidArgumentError); + // @ts-expect-error deliberate invalid policy + expect(() => message({ aggregateHash: 'sideways' })).toThrow( + /unknown aggregateHash 'sideways', expected one of: auto, none, header, footer, both/, + ); + }); +}); + +describe('AppendOptions mirror the encode knobs', () => { + let tmp: string; + beforeEach(() => { + tmp = mkdtempSync(join(tmpdir(), 'tensogram-ts-aggregate-')); + }); + afterEach(() => { + rmSync(tmp, { recursive: true, force: true }); + }); + + it('forwards aggregateHash through TensogramFile#append', async () => { + await init(); + const path = join(tmp, 'append.tgm'); + writeFileSync(path, message()); + + const file = await TensogramFile.open(path); + try { + await file.append( + defaultMeta(), + [{ descriptor: makeDescriptor([4], 'float32'), data: new Float32Array([9, 9, 9, 9]) }], + { aggregateHash: 'both' }, + ); + expect(file.messageCount).toBe(2); + // The seed message used the default policy; only the appended one + // must carry the footer aggregate. + const seed = await file.rawMessage(0); + const appended = await file.rawMessage(1); + expect(frameTypes(seed)).not.toContain('FooterHash'); + expect(frameTypes(appended)).toContain('HeaderHash'); + expect(frameTypes(appended)).toContain('FooterHash'); + } finally { + file.close(); + } + }); + + it('rejects an unknown appended aggregateHash policy', async () => { + await init(); + const path = join(tmp, 'reject.tgm'); + writeFileSync(path, message()); + const file = await TensogramFile.open(path); + try { + await expect( + file.append( + defaultMeta(), + [{ descriptor: makeDescriptor([4], 'float32'), data: new Float32Array([1, 2, 3, 4]) }], + // @ts-expect-error deliberate invalid policy + { aggregateHash: 'sideways' }, + ), + ).rejects.toThrow(InvalidArgumentError); + } finally { + file.close(); + } + }); +}); + +describe('EncodeOptions.compressionBackend', () => { + /** + * A zstd-compressed message — the only kind whose bytes a real backend + * switch could change. Returns the compressed data-object payloads so + * the comparison ignores the per-message provenance (UUID / timestamp) + * that lives in the metadata frame. + */ + function dataObjectPayloads( + compression: 'none' | 'zstd', + compressionBackend?: CompressionBackend, + ): number[][] { + const descriptor = { ...makeDescriptor([64], 'float32'), compression }; + const data = new Float32Array(64).map((_, i) => i % 4); + const opts: EncodeOptions = compressionBackend ? { compressionBackend } : {}; + const msg = encode(defaultMeta(), [{ descriptor, data }], opts); + return frames(msg) + .filter((f) => f.frameType === 'NTensorFrame') + .map((f) => Array.from(f.payload)); + } + + it('accepts every backend name and is a no-op on WASM', async () => { + await init(); + // WASM ships pure-Rust codecs only (lz4 / szip-pure / zstd-pure), so + // no value can change the codec that actually runs. Compare the + // data-object payload BYTES — the message as a whole also carries + // per-encode provenance (uuid / timestamp) that always differs. + const auto = dataObjectPayloads('zstd', 'auto'); + // Guard against a vacuous comparison: the codec really did run. + expect(auto[0].length).toBeLessThan(dataObjectPayloads('none')[0].length); + expect(dataObjectPayloads('zstd', 'pure')).toEqual(auto); + expect(dataObjectPayloads('zstd', 'ffi')).toEqual(auto); + expect(dataObjectPayloads('zstd')).toEqual(auto); + }); + + it('rejects an unknown backend name', async () => { + await init(); + // @ts-expect-error deliberate invalid backend + expect(() => message({ compressionBackend: 'cuda' })).toThrow(InvalidArgumentError); + // @ts-expect-error deliberate invalid backend + expect(() => message({ compressionBackend: 'cuda' })).toThrow( + /unknown compressionBackend 'cuda', expected one of: auto, ffi, pure/, + ); + }); +}); diff --git a/typescript/tests/frameWalk.test.ts b/typescript/tests/frameWalk.test.ts new file mode 100644 index 00000000..9a8af109 --- /dev/null +++ b/typescript/tests/frameWalk.test.ts @@ -0,0 +1,300 @@ +// (C) Copyright 2026- ECMWF and individual contributors. +// +// This software is licensed under the terms of the Apache Licence Version 2.0 +// which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. +// +// Tests for the frame walker (`frames`) and the typed message envelope +// (`messageHeader`). Mirror the Rust core `tensogram::frames` / +// `tensogram::message_header` (`rust/tensogram/src/frame_walk.rs`). + +import { describe, expect, it } from 'vitest'; +import { + FramingError, + InvalidArgumentError, + StreamingEncoder, + WIRE_VERSION, + encode, + frames, + init, + messageHeader, + scan, +} from '../src/index.js'; +import type { Frame, FrameTypeName } from '../src/index.js'; +import { defaultMeta, makeDescriptor } from './helpers.js'; + +/** Bytes in the fixed message preamble — never a frame. */ +const PREAMBLE_BYTES = 24; +/** Bytes in the fixed message postamble — never a frame. */ +const POSTAMBLE_BYTES = 24; +/** Bytes in every frame header. */ +const FRAME_HEADER_BYTES = 16; +/** Frame-type-specific footer sizes: 20 B for type 9, 12 B otherwise. */ +function footerBytes(frameTypeCode: number): number { + return frameTypeCode === 9 ? 20 : 12; +} + +/** A buffered (random-access) message with `n` float32 objects. */ +function bufferedMessage(n: number): Uint8Array { + const objects = Array.from({ length: n }, (_, i) => ({ + descriptor: makeDescriptor([4], 'float32'), + data: new Float32Array([i, i + 1, i + 2, i + 3]), + })); + return encode(defaultMeta(), objects); +} + +/** A streaming-mode message (footer-side index / hashes). */ +function streamedMessage(): Uint8Array { + const enc = new StreamingEncoder(defaultMeta()); + try { + enc.writeObject(makeDescriptor([4], 'float32'), new Float32Array([1, 2, 3, 4])); + return enc.finish(); + } finally { + enc.close(); + } +} + +function typesOf(fs: readonly Frame[]): FrameTypeName[] { + return fs.map((f) => f.frameType); +} + +describe('frames', () => { + it('walks a buffered message in wire order', async () => { + await init(); + const msg = bufferedMessage(2); + // Buffered (random-access) layout, default options: metadata, index + // and the aggregate hash all live in the header; the two data + // objects follow. The preamble / postamble are NOT frames. + expect(typesOf(frames(msg))).toEqual([ + 'HeaderMetadata', + 'HeaderIndex', + 'HeaderHash', + 'NTensorFrame', + 'NTensorFrame', + ]); + }); + + it('yields one data-object frame per encoded object', async () => { + await init(); + for (const n of [1, 3, 5]) { + const fs = frames(bufferedMessage(n)); + expect(fs.filter((f) => f.frameType === 'NTensorFrame')).toHaveLength(n); + } + }); + + it('pairs every frameType with its wire frameTypeCode', async () => { + await init(); + const codes: Record = { + HeaderMetadata: 1, + HeaderIndex: 2, + HeaderHash: 3, + FooterHash: 5, + FooterIndex: 6, + FooterMetadata: 7, + PrecederMetadata: 8, + NTensorFrame: 9, + }; + for (const f of frames(bufferedMessage(2))) { + expect(f.frameTypeCode).toBe(codes[f.frameType]); + } + for (const f of frames(streamedMessage())) { + expect(f.frameTypeCode).toBe(codes[f.frameType]); + } + }); + + it('reports offsets and lengths that are in bounds and never overlap', async () => { + await init(); + const msg = bufferedMessage(3); + let prevEnd = PREAMBLE_BYTES; + for (const f of frames(msg)) { + // Offsets are relative to the message start, after the preamble. + expect(f.offset).toBeGreaterThanOrEqual(prevEnd); + expect(f.length).toBeGreaterThanOrEqual(FRAME_HEADER_BYTES); + expect(f.offset + f.length).toBeLessThanOrEqual(msg.byteLength - POSTAMBLE_BYTES); + prevEnd = f.offset + f.length; + } + // At least one frame was walked, so prevEnd advanced. + expect(prevEnd).toBeGreaterThan(PREAMBLE_BYTES); + }); + + it('returns payload with the frame header and type footer stripped', async () => { + await init(); + const msg = bufferedMessage(2); + for (const f of frames(msg)) { + const footer = footerBytes(f.frameTypeCode); + expect(f.payload.byteLength).toBe(f.length - FRAME_HEADER_BYTES - footer); + // The payload is exactly the frame's content slice. + const start = f.offset + FRAME_HEADER_BYTES; + expect(Array.from(f.payload)).toEqual( + Array.from(msg.subarray(start, start + f.payload.byteLength)), + ); + // The stripped tail always ends with the ENDF magic. + const end = msg.subarray(f.offset + f.length - 4, f.offset + f.length); + expect(new TextDecoder().decode(end)).toBe('ENDF'); + // ... and the frame header starts with the FR magic. + const head = msg.subarray(f.offset, f.offset + 2); + expect(new TextDecoder().decode(head)).toBe('FR'); + } + }); + + it('copies payload onto the JS heap (independent of the source buffer)', async () => { + await init(); + const msg = bufferedMessage(1); + const fs = frames(msg); + const first = fs[0]; + const before = first.payload[0]; + // Mutating the copy must not touch the caller's buffer ... + first.payload[0] = (before ^ 0xff) & 0xff; + expect(msg[first.offset + FRAME_HEADER_BYTES]).toBe(before); + // ... and the copy survives further WASM calls (memory growth safe). + frames(bufferedMessage(64)); + expect(first.payload[0]).toBe((before ^ 0xff) & 0xff); + expect(first.payload.buffer).not.toBe(msg.buffer); + }); + + it('exposes hasHash mirroring the per-frame HASH_PRESENT flag', async () => { + await init(); + const hashed = frames(bufferedMessage(1)); + expect(hashed.some((f) => f.hasHash)).toBe(true); + for (const f of hashed) { + expect(f.hasHash).toBe((f.flags & 0b10) !== 0); + } + const unhashed = frames( + encode( + defaultMeta(), + [{ descriptor: makeDescriptor([4], 'float32'), data: new Float32Array([1, 2, 3, 4]) }], + { hash: false }, + ), + ); + expect(unhashed.every((f) => !f.hasHash)).toBe(true); + }); + + it('reports the wire version on every frame', async () => { + await init(); + for (const f of frames(bufferedMessage(1))) { + expect(f.version).toBe(1); + } + }); + + it('walks a streaming message including its footer frames', async () => { + await init(); + const types = typesOf(frames(streamedMessage())); + expect(types[0]).toBe('HeaderMetadata'); + expect(types).toContain('NTensorFrame'); + // Streaming defers index / metadata to the footer region; the walk + // must reach them even though `total_length` may never have been + // back-filled. + expect(types).toContain('FooterIndex'); + }); + + it('throws on a buffer that is not a message', async () => { + await init(); + // Too short for a preamble, empty, and long-enough-but-wrong-magic. + expect(() => frames(new Uint8Array([1, 2, 3, 4]))).toThrow(FramingError); + expect(() => frames(new Uint8Array(0))).toThrow(FramingError); + expect(() => frames(new Uint8Array(64))).toThrow(/magic/); + }); + + it('throws on a message with a truncated frame chain', async () => { + await init(); + const msg = bufferedMessage(2); + const fs = frames(msg); + const last = fs[fs.length - 1]; + // Cut inside the last frame: the preamble still parses, the chain + // does not. An eagerly-materialised array is all-or-nothing — a + // corrupt chain must never surface as a silently shorter array. + expect(() => frames(msg.subarray(0, last.offset + 8))).toThrow(FramingError); + }); + + it('throws on non-Uint8Array input', async () => { + await init(); + // @ts-expect-error deliberate wrong type + expect(() => frames('nope')).toThrow(InvalidArgumentError); + // @ts-expect-error deliberate wrong type + expect(() => frames('nope')).toThrow(/Uint8Array/); + }); + + it('walks each message of a multi-message buffer after scan() + slice', async () => { + await init(); + const a = bufferedMessage(1); + const b = bufferedMessage(3); + const buf = new Uint8Array(a.byteLength + b.byteLength); + buf.set(a, 0); + buf.set(b, a.byteLength); + + const positions = scan(buf); + expect(positions).toHaveLength(2); + const counts = positions.map((p) => { + const slice = buf.subarray(p.offset, p.offset + p.length); + return frames(slice).filter((f) => f.frameType === 'NTensorFrame').length; + }); + expect(counts).toEqual([1, 3]); + }); +}); + +describe('messageHeader', () => { + it('reports the wire version and total length of a buffered message', async () => { + await init(); + const msg = bufferedMessage(2); + const h = messageHeader(msg); + expect(h.version).toBe(WIRE_VERSION); + expect(h.totalLength).toBe(msg.byteLength); + }); + + it('flags match the frames present exactly in buffered mode', async () => { + await init(); + const msg = bufferedMessage(2); + const h = messageHeader(msg); + const types = new Set(typesOf(frames(msg))); + expect(h.hasHeaderMetadata).toBe(types.has('HeaderMetadata')); + expect(h.hasFooterMetadata).toBe(types.has('FooterMetadata')); + expect(h.hasHeaderIndex).toBe(types.has('HeaderIndex')); + expect(h.hasFooterIndex).toBe(types.has('FooterIndex')); + expect(h.hasHeaderHashes).toBe(types.has('HeaderHash')); + expect(h.hasFooterHashes).toBe(types.has('FooterHash')); + expect(h.hasPrecederMetadata).toBe(types.has('PrecederMetadata')); + }); + + it('reports hasHashesPresent following the hash option', async () => { + await init(); + expect(messageHeader(bufferedMessage(1)).hasHashesPresent).toBe(true); + const unhashed = encode( + defaultMeta(), + [{ descriptor: makeDescriptor([4], 'float32'), data: new Float32Array([1, 2, 3, 4]) }], + { hash: false }, + ); + expect(messageHeader(unhashed).hasHashesPresent).toBe(false); + }); + + it('never understates the frames present in streaming mode', async () => { + await init(); + const msg = streamedMessage(); + const h = messageHeader(msg); + expect(h.version).toBe(WIRE_VERSION); + const types = new Set(typesOf(frames(msg))); + // The streaming encoder writes the preamble before any object, so + // its flags are advisory: only `frame present => flag set` holds. + const implies = (present: boolean, flag: boolean): boolean => !present || flag; + expect(implies(types.has('HeaderMetadata'), h.hasHeaderMetadata)).toBe(true); + expect(implies(types.has('FooterMetadata'), h.hasFooterMetadata)).toBe(true); + expect(implies(types.has('HeaderIndex'), h.hasHeaderIndex)).toBe(true); + expect(implies(types.has('FooterIndex'), h.hasFooterIndex)).toBe(true); + expect(implies(types.has('HeaderHash'), h.hasHeaderHashes)).toBe(true); + expect(implies(types.has('FooterHash'), h.hasFooterHashes)).toBe(true); + expect(implies(types.has('PrecederMetadata'), h.hasPrecederMetadata)).toBe(true); + }); + + it('throws on a buffer that is not a message', async () => { + await init(); + expect(() => messageHeader(new Uint8Array([1, 2, 3, 4]))).toThrow(FramingError); + expect(() => messageHeader(new Uint8Array(0))).toThrow(FramingError); + expect(() => messageHeader(new Uint8Array(64))).toThrow(/magic/); + }); + + it('throws on non-Uint8Array input', async () => { + await init(); + // @ts-expect-error deliberate wrong type + expect(() => messageHeader(123)).toThrow(InvalidArgumentError); + // @ts-expect-error deliberate wrong type + expect(() => messageHeader(123)).toThrow(/Uint8Array/); + }); +});