feat: cross-language symmetry — Wave A (bugs + cheap [O] omissions across all bindings)

[tlmquintino]

First implementation wave of the cross-language interface symmetry roadmap (plans/INTERFACE_SYMMETRY.md §8, docs in #163). Closes the confirmed defects and the cheap [O] omissions — capabilities the backend already provides that a binding simply never exposed. Reference = Rust core. Each lane is disjoint by binding and self-verified.

C ABI

• BUG-FFI — implement tgm_last_error_object_index() (was documented in the TGM_ERROR_MISSING_HASH note but undefined). Thread-local object index recorded alongside the last error; header regenerated. The MissingHash/HashMismatch offending object is now reachable from C (and C++ — see below). 198 ffi tests.

Python

• BUG-PY — object_inline_hashes(buf) (binds core data_object_inline_hashes) makes v3 inline hashes readable; fixed the dangling DataObjectDescriptor.hash docstring.
• AsyncStreamingEncoder; StreamingEncoder introspection (object_count/bytes_written/write_preceder); compute_common. +42 tests → 693 passed.

TypeScript

• BUG-TS — TensogramFile.append now forwards the full AppendOptions (mask options were silently dropped).
• TensogramFile.create; native_byte_order decode option (minimal wasm widen, existing decode byte-identical); export DecodeStreamOptions. +17 tests → 454 passed.

Fortran

• Bind 23 previously-omitted synchronous C-ABI functions (49→72): TGM_WIRE_VERSION + version getters, doctor, validate/validate_file, scan, compute_hash + inline-hash accessors, object byte_order/type/filter/compression/encoding/strides, encode_pre_encoded, decode_object/range/metadata. All [O]; no language-limited items. 105-assertion test; ctest 26/26; f2008 -Werror clean.

C++

• doctor() (whole category was missing); typed compute_simple_packing_params(); async_file::decode_object/decode_range on all three async frontends; async path() + pull-model task<T> (ready()/cancel()/join()); integrity_error::object_index() sourced from the new C ABI symbol. +31 tests → cpp ctest 241/241.

Notes

• Every new capability has tests and at least one example in its language (symmetry Examples contract).
• Remaining roadmap (Wave B C-ABI widening for the [B] backend gaps — decode_with_masks, scan_file/ScanOptions, validate_buffer, typed enums, wire introspection, sync remote — then Wave C C++/Fortran wrappers, Wave E convert) is tracked in plans/INTERFACE_SYMMETRY.md.

Docs Preview
https://sites.ecmwf.int/docs/tensogram/pull-requests/PR-164

[Copilot]

Pull request overview

Implements “Wave A” of the cross-language interface symmetry roadmap by fixing confirmed binding defects and exposing previously-omitted (already-supported) capabilities across C ABI, Python, TypeScript/WASM, Fortran, and C++, with accompanying tests and examples to keep surfaces aligned with the Rust core reference.

Changes:

• Adds missing/buggy binding surface area (notably C ABI last-error object index, TS append option forwarding + file factory, Python inline-hash access + compute helpers, C++ doctor/async decode helpers).
• Widens decode/encode option plumbing (e.g., TS nativeByteOrder) and improves structured error reporting (object index for integrity failures).
• Adds new cross-language examples and expanded test coverage validating the new APIs end-to-end.

Reviewed changes

Copilot reviewed 43 out of 43 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
typescript/tests/streaming.test.ts	Adds a regression test ensuring DecodeStreamOptions is nameable via index re-export.
typescript/tests/nativeByteOrder.test.ts	Adds tests covering DecodeOptions.nativeByteOrder behavior and forwarding through TensogramFile.message.
typescript/tests/create.test.ts	Adds tests for TensogramFile.create workflows (truncate, mkdir parents, URL support, invalid args).
typescript/tests/append.test.ts	Adds regression tests proving append() forwards full mask/NaN/Inf options to encode().
typescript/src/types.ts	Adds DecodeOptions.nativeByteOrder documentation and option surface.
typescript/src/index.ts	Re-exports DecodeStreamOptions type.
typescript/src/file.ts	Implements TensogramFile.create, adds node write helpers, and fixes append() option forwarding.
typescript/src/decode.ts	Plumbs nativeByteOrder through to the wasm decode functions.
rust/tensogram-wasm/src/lib.rs	Widens wasm decode/decode_object exports to accept native_byte_order.
rust/tensogram-ffi/tensogram.h	Declares the new tgm_last_error_object_index() C ABI function.
rust/tensogram-ffi/src/lib.rs	Implements thread-local last-error object index tracking + adds a regression test.
python/tests/test_streaming_encoder_introspection.py	Tests new StreamingEncoder introspection + write_preceder behavior.
python/tests/test_object_inline_hashes.py	Tests tensogram.object_inline_hashes() (v3 inline hash accessibility).
python/tests/test_compute_common.py	Tests tensogram.compute_common() binding and edge cases.
python/tests/test_async_streaming_encoder.py	Tests the new AsyncStreamingEncoder (async writes, finish modes, introspection, errors).
python/bindings/src/lib.rs	Adds Python bindings for inline hashes, compute_common, streaming encoder introspection, and async streaming encoder.
python/bindings/python/tensogram/init.py	Adds explicit Python re-exports for compute_common / object_inline_hashes.
python/bindings/Cargo.toml	Extends async feature dependencies to include tokio (sync mutex).
fortran/test/test_symmetry.f90	Adds a broad symmetry test exercising newly exposed Fortran procedures.
fortran/src/tensogram.F90	Adds Wave-A Fortran bindings: version, doctor/validate/scan/hash, descriptor/hash accessors, pre-encoded encode, decode variants.
fortran/CMakeLists.txt	Wires in the new Fortran symmetry test and validate example.
examples/typescript/README.md	Documents new TypeScript example for create + mask options.
examples/typescript/package.json	Adds script for the new TS example and includes it in all.
examples/typescript/19_file_create_masks.ts	Demonstrates TensogramFile.create and mask-option append round-trip.
examples/python/README.md	Documents new Python example and newly exposed API surface items.
examples/python/19_async_streaming_and_common.py	Demonstrates async streaming encode, compute_common, and object_inline_hashes.
examples/fortran/validate.f90	Adds Fortran example for scan/version/validate workflows.
examples/fortran/README.md	Documents the new Fortran validate example.
examples/cpp/README.md	Documents new C++ doctor and async object/range example.
examples/cpp/CMakeLists.txt	Adds new C++ examples to the build.
examples/cpp/25_async_decode_object_range.cpp	Demonstrates async decode_object/decode_range + task handle/path accessors.
examples/cpp/10_doctor.cpp	Demonstrates tensogram::doctor() reporting.
cpp/tests/test_simple_packing.cpp	Adds tests for typed compute_simple_packing_params() helper.
cpp/tests/test_doctor.cpp	Adds tests for tensogram::doctor() wrapper output shape/stability.
cpp/tests/test_decode_verify_hash.cpp	Adds tests for structured integrity-error object index accessor.
cpp/tests/test_async_task_control.cpp	Adds tests for async path accessors and pull-model task control (ready/cancel/join).
cpp/tests/test_async_decode_object_range.cpp	Adds tests for async decode_object/decode_range in callback + std::future frontends.
cpp/tests/test_async_coro.cpp	Adds coroutine-frontend tests for decode_object/decode_range and task/path access.
cpp/tests/CMakeLists.txt	Registers new C++ tests in the build.
cpp/include/tensogram/async/std_future.hpp	Exposes async decode_object/decode_range and pull-model tasks + path for std::future frontend.
cpp/include/tensogram/async/coro.hpp	Exposes async decode_object/decode_range and pull-model tasks + path for coroutine frontend.
cpp/include/tensogram/async/callback.hpp	Adds async range decode buffer handling + pull-model task<T> API + new decode/path methods.
cpp/include/tensogram.hpp	Adds doctor wrapper, typed simple packing params helper, and structured integrity error object_index plumbing.

Comments suppressed due to low confidence (1)

cpp/include/tensogram/async/callback.hpp:695

• Same empty-ranges issue as the callback-based decode_range: offsets.data() / counts.data() may be null for empty vectors, but the FFI requires non-null pointers regardless of n_ranges. Ensure non-null pointers are passed even when ranges.size() == 0.

        tensogram::detail::check(tgm_async_file_decode_range(
            handle_.get(), msg_index, obj_index,
            offsets.empty() ? nullptr : offsets.data(),
            counts.empty() ? nullptr : counts.data(),
            ranges.size(), true, 0,
            token ? token->raw() : nullptr,
            static_cast<uint64_t>(timeout.count()), &raw));
        return task<detail::range_buffers>(raw, &detail::join_multi_bytes_result);

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 69 out of 69 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

typescript/src/file.ts:228

• The error message says "string or file:// URL", but this function only treats file:// as a URL object (a "file://..." string would be passed to fs.writeFile as a literal path and fail). Consider clarifying the message to explicitly say "file:// URL object" (or alternatively add support for file-URL strings).