feat: cross-language symmetry — deferred Wave B (frame walker, sync remote, typed enums) - #167
Open
tlmquintino wants to merge 9 commits into
Open
feat: cross-language symmetry — deferred Wave B (frame walker, sync remote, typed enums)#167tlmquintino wants to merge 9 commits into
tlmquintino wants to merge 9 commits into
Conversation
Deferred Wave B, component A0 — the reference implementation every binding
mirrors (plans/INTERFACE_SYMMETRY.md §9).
- frames(message) -> FrameIter: lazy, per-message walk of the type 1-9 frames.
Each FrameInfo carries {frame_type, version, flags, offset, length} plus a
borrowed content payload (frame header + type-specific footer stripped);
offset is relative to the message start. has_hash() exposes the per-frame
HASH_PRESENT bit.
- message_header(message) -> MessageHeader: the preamble as typed values
(version, total_length) plus the eight structural predicates.
- The walk is bounded to the frame region so the postamble is never mistaken
for a frame, including streaming messages whose total_length was never
back-filled (detected via the trailing END_MAGIC).
TDD: tests written first (red), then implemented. 8 tests; core suite 746.
Deferred Wave B, component A1 — wrap the core frame walker in the C ABI:
- tgm_frame_type enum (1-9, 4 reserved) + cbindgen rename.
- TgmFrame POD {frame_type, version, flags, offset, length, payload, payload_len}
and TgmMessageHeader POD (version, total_length + 8 typed booleans).
- Lazy iterator tgm_frame_iter_create/_next/_free (opaque handle borrowing the
caller's message) + tgm_frame_has_hash + tgm_message_header.
- Lifetime contract documented on the handle, create, and TgmFrame: payload
points INTO the caller's msg and outlives the iterator; msg must outlive it.
- tgm_frame_iter_next clears the sticky last-error on clean exhaustion so
callers can distinguish 'end' from 'malformed' via tgm_last_error.
- New check_frame_type_enum.sh guard (C header vs core FrameType) wired into
the Fortran ctest suite alongside the error/value-type guards.
TDD: 3 red->green cycles, tests written first. +18 tests (229 -> 247); clippy
clean; header regenerated with no cargo-c drift.
…(A4 + C)
Deferred Wave B, Python lane:
- frames(buf) -> FrameIter yielding Frame {frame_type (name), frame_type_code,
version, flags, offset, length, payload, has_hash, is_data_object}.
- message_header(buf) -> MessageHeader (version, total_length, flags + the
eight has_* predicates, keeping the core predicate names).
- compression_backend= ('auto'|'ffi'|'pure') on encode / append /
StreamingEncoder / AsyncStreamingEncoder. Note: aggregate_hash= turned out to
be already wired; it is now covered by tests and documented.
TDD: tests written first (81 failed red -> 94 green), plus a second red cycle
for FrameIter.__len__. +96 tests; suite 867 passed / 52 skipped; ruff clean.
Deferred Wave B, TypeScript/WASM lane: - frames(buf) -> Frame[] and messageHeader(buf) -> MessageHeader via a new wasm frame_walk module; Frame payloads are copied into the JS heap (a view would dangle — wasm-bindgen frees the argument allocation on return). - Returns an array, not a generator: the wasm call materialises the whole walk, so a malformed chain throws rather than silently yielding fewer frames. - EncodeOptions/AppendOptions gain aggregateHash (forwarded) and compressionBackend (validated, documented as a WASM platform no-op — pure-Rust codecs only, mirroring the accepted [L] exception for threads). TDD: tests written first (27 failed red -> green), plus a second red cycle for append forwarding. +33 vitest (486 -> 519) and +10 wasm tests (183 -> 193).
Deferred Wave B, components B and C.
B — synchronous remote (new opt-in 'remote' feature; does NOT imply async):
- tgm_is_remote_url, TgmRemoteScanOptions {bidirectional},
tgm_file_open_remote(source, keys[], values[], n, opts, out) returning a plain
tgm_file_t so the existing sync file API works on remote sources.
- Argument validation runs BEFORE the feature check, so every build answers
INVALID_ARG identically; feature-off stubs return TGM_ERROR_REMOTE with a
message naming --features=remote. Symbols always link.
C — typed enums + encode options:
- tgm_dtype (15), tgm_byte_order, tgm_aggregate_hash_policy,
tgm_compression_backend; each enum's zero value is the library default so a
zero-initialised C struct behaves correctly.
- tgm_object_dtype_enum / tgm_object_byte_order_enum (string getters unchanged).
- TgmEncodeOptions POD (hash + aggregate_hash + compression_backend + mask
fields) with *_with_encode_options entry points; TgmEncodeMaskOptions and all
existing entry points untouched.
- Drift protection: exhaustive-match Rust tests (a new core variant breaks the
build) plus check_dtype_enum.sh for the checked-in generated header.
Closed loop: aggregate_hash=BOTH verified via the new frame walker (both
HEADER_HASH and FOOTER_HASH frames present). TDD: 7 red->green cycles.
+34 tests (247 -> 281 all-features); clippy clean; header byte-identical across
default/remote/all-features; no cargo-c drift.
…2 + B + C) Deferred Wave B, C++ lane — wrap the widened C ABI: - A2: enum class frame_type, a value-type frame, and a lazy frame_range with an input iterator + C++17 sentinel end(); frames(msg,len) and read_message_header(). The range owns the cursor (RAII, movable without invalidating live iterators); payload borrows the caller's buffer and stays valid after the range dies (asserted by test). A false from the C iterator is disambiguated via tgm_last_error: clean end stops the loop, malformed throws framing_error. - B: is_remote_url + file::open_remote(source, storage_options, opts) returning a plain file. Adds NUL-embedding validation C++ can actually get wrong (a truncated C string would silently open a different object). Wired TENSOGRAM_REMOTE into cpp/CMakeLists (default OFF); verified both states. - C: enum class dtype/byte_order/aggregate_hash_policy/compression_backend, decoded_object::dtype_enum()/byte_order_enum() (throwing on OOB rather than returning the indistinguishable zero variant), and encode_options gaining aggregate_hash + codec_backend. Deliberate deviation: no *_with_encode_options C++ overloads — they would be signature-identical to the existing entry points. Those now escalate to the new C functions when the new fields are set, matching the header's existing three-tier ladder. encode_pre_encoded throws instead of emitting a message that contradicts options the C ABI cannot honour there. TDD: 5 red->green cycles (incl. one runtime red). +57 tests; cpp ctest 336/336, and 59/59 of the new tests with -DTENSOGRAM_REMOTE=ON (real file:// round-trip).
…s (A3 + B + C) Deferred Wave B, Fortran lane — bind the widened C ABI: - A3: tensogram_frame (value type; fields and payload bytes copied out), tensogram_frame_iterator (%next/%free, non-copyable + final), and tensogram_message_header (version/total_length + the eight has_*), via tensogram_frames() and tensogram_message_header_read(). The C cursor borrows its buffer for its whole life, which no Fortran dummy can promise (a non-contiguous actual dies with its temporary), so the iterator copies the buffer and lends the cursor that — sound by construction, and callers need no target attribute. %offset() is 1-based, matching tensogram_scan. End vs malformed is disambiguated through the optional err (OK vs FRAMING), with INVALID_ARG for a freed cursor. - B: tensogram_is_remote_url and a generic tensogram_file_open_remote (with or without storage-option key/value arrays — 'keys without values' is a compile-time error), returning a plain tensogram_file. - C: tensogram_object_dtype_enum/_byte_order_enum (string getters untouched) and *_with_options entry points exposing every TgmEncodeOptions knob as optional arguments; TGM_DTYPE_*/BYTE_ORDER_*/AGGREGATE_HASH_POLICY_*/ COMPRESSION_BACKEND_*/FRAME_TYPE_* params. - One parameterised check_enum_mirror.sh guard registered 5x replaces five near-identical scripts. TDD: 5 red->green cycles plus mutation teeth-checks (break the source, watch the right test fail, restore). +11 ctest entries (30 -> 41), 401 assertions; valgrind clean; f2008 -Werror clean; test_remote re-run against a --features remote C library gives 28/28 with a live file:// round-trip.
New docs/src/guide/frame-introspection.md (wired into SUMMARY.md beside the metadata guide): the language-neutral contract — what counts as a frame (types 1-3 and 5-9; 4 is reserved), preamble/postamble are not frames, the one-message rule with the scan-then-slice pattern, offsets/spans/payload boundaries (20 B vs 12 B footer), the per-binding borrow-vs-copy lifetime table, the buffered-exact vs streaming-advisory header-flag rule, end-of-walk vs malformed chain, and a snippet per language. Per-language guides updated: c-api.md gains the opt-in "remote" feature (with the not-in-the-published-tarball caveat), frame introspection, and the typed enums / TgmEncodeOptions; cpp/python/typescript/fortran guides gain their corresponding sections. CHANGELOG [Unreleased] records all three capabilities. Notable corrections found while writing: streaming also emits a HeaderMetadata frame, so has_header_index (not has_header_metadata) discriminates random-access; Python frames() walks eagerly and materialises lazily; TS aggregateHash/compressionBackend exist on encode/append only.
Closes the last example gap for the frame walker (C++/Python/TS/Fortran already had one), per the symmetry Examples contract. Reads the envelope, walks the frames of a buffered vs a streaming message, and verifies aggregate_hash placement with the walker itself. Also demonstrates two documented nuances: has_header_index (not has_header_metadata) discriminates random access, and a streaming message reports preceder=true with no PrecederMetadata frame.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the deferred Wave-B plan (
plans/INTERFACE_SYMMETRY.md§9). Stacked on #164 — GitHub will retarget this tomainwhen that merges. Built strictly TDD: every capability's test was written and observed failing before its implementation.A · Frame walker + typed message header — all six bindings
Structural introspection of one message: walk its frames and read its envelope without decoding payloads or CBOR.
frames(msg) -> FrameIter(lazy) yieldingFrameInfo { frame_type, version, flags, offset, length, payload }+has_hash();message_header(msg) -> MessageHeader(version,total_length, eight typed predicates).tgm_frame_type,TgmFrame/TgmMessageHeader,tgm_frame_iter_create/_next/_free,tgm_frame_has_hash,tgm_message_header.enum class frame_type,frame, lazyframe_range(range-for via a C++17 sentinel),frames(),read_message_header().frames()/message_header(),Frame,FrameIter,MessageHeader.frames(),messageHeader()(new wasmframe_walkmodule).tensogram_frames(),tensogram_frame_iterator,tensogram_frame,tensogram_message_header_read().Contract: only the type 1–3/5–9 frames are walked (4 is reserved; the preamble/postamble are not frames — use the header).
offsetis message-relative (1-based in Fortran, 0-based elsewhere);payloadis frame content (16-byte header and 20 B/12 B type-specific footer stripped). C/C++ borrow the caller's buffer; Python/TS/Fortran copy. Afalse/end from the C iterator is disambiguated viatgm_last_error(clean end vs malformed).B · Synchronous remote access — C, C++, Fortran
Previously only the async C API could open a remote
.tgm. Nowtgm_is_remote_url+tgm_file_open_remote(source, keys[], values[], n, opts, out)returns a plain sync file handle, so the existing file API works unchanged on S3/HTTP sources; mirrored as C++file::open_remoteand Fortrantensogram_file_open_remote.Opt-in
remoteCargo feature — deliberately NOT in the default published C-API tarball (same model asasync); the header declares the symbols either way and a feature-off build returnsTGM_ERROR_REMOTEnaming--features=remote. C++ exposes-DTENSOGRAM_REMOTE=ON(default OFF). Both C++ and Fortran verified the key/value marshalling against a livefile://backend with the feature on.C · Typed enums + full encode options
tgm_dtype(15),tgm_byte_order,tgm_aggregate_hash_policy,tgm_compression_backend(each enum's zero value is the library default, so a zero-initialised C struct is correct),tgm_object_dtype_enum/_byte_order_enum(string getters unchanged), andTgmEncodeOptionsexposing the previously-unreachableaggregate_hashandcompression_backend. Python gainedcompression_backend=; TS gainedaggregateHash(and documentscompressionBackendas a WASM no-op — pure-Rust codecs only).Drift protection: exhaustive-match Rust tests (a new core variant breaks the build) plus header/Fortran mirror guards, wired into ctest.
Notable design calls made during implementation
*_with_encode_optionsoverloads — they'd be signature-identical to the existing entry points. Those now escalate to the new C functions when the new fields are set.encode_pre_encodedthrows rather than emitting a message contradicting options the C ABI cannot honour there.targetattribute.Frame.payloadis copied into the JS heap — a view would dangle, since wasm-bindgen frees the argument allocation on return.Finding surfaced by TDD (not fixed here)
The streaming encoder sets the
PRECEDER_METADATApreamble flag unconditionally and advisorily (it cannot know at preamble-write time whether preceders follow) and may leavetotal_lengthat 0 — butplans/WIRE_FORMAT.md§3.1 states that flag definitively. Implementation and spec disagree. The implementation is deliberate and documented in code, so I documented the actual behaviour in the guide and left the wire spec untouched — it needs a spec-owner decision. Tests encode the true contract: buffered messages match exactly, streaming only guaranteesframe present ⇒ flag set.Docs
New
docs/src/guide/frame-introspection.md(language-neutral contract + a snippet per language), sections added to the c/cpp/python/typescript/fortran API guides (the C-API guide calls out the opt-inremotecaveat prominently), a Rust runnable example closing the last example gap, and aCHANGELOG [Unreleased]entry.Verification (full matrix, all green)
Rust core 746 · workspace 69 suites, 0 failures · FFI 272 / 273 (remote) / 281 (all-features) · C++ ctest 336/336 (+59 with remote ON) · Fortran ctest 41/41, f2008
-Werrorclean, valgrind clean · TS 519 vitest · wasm 193 · Python 867 passed / 52 skipped ·cargo-c-header-checkno drift (header byte-identical across feature sets) · clippy 0 warnings · rust-fmt clean ·make docs-buildclean.Docs Preview
https://sites.ecmwf.int/docs/tensogram/pull-requests/PR-167