docs: cross-language interface symmetry — goal, gap analysis, and roadmap - #163
Open
tlmquintino wants to merge 5 commits into
Open
docs: cross-language interface symmetry — goal, gap analysis, and roadmap#163tlmquintino wants to merge 5 commits into
tlmquintino wants to merge 5 commits into
Conversation
A deep audit of all six language surfaces (Rust reference, C FFI, C++, Python, TypeScript/WASM, Fortran) — interface code, the backend each binds, and example coverage — found substantial, undocumented asymmetry. Document the goal of full symmetry and the audit results: - AGENTS.md: a required "Cross-language interface symmetry" rule (Rust core is the reference; mirror features across all bindings; classify gaps as [O]mission / [B]ackend-gap / [L]anguage-limit; widen the C ABI to unblock the C-family; verify against code not docs) and a stronger Examples contract (cover the full public surface per language). - plans/DESIGN.md: new "Cross-Language Interface Symmetry" section — the goal, why the C ABI is the bottleneck for C/C++/Fortran, the gap taxonomy, the audited 24x6 feature matrix, the backend gaps to close first, the accepted [L] language-limit exceptions (Fortran async/half-float/unsigned/JSON-builder; WASM threads), and the concrete interface defects the audit surfaced.
New plans/INTERFACE_SYMMETRY.md capturing the 0.24.0 audit: method, why the surface diverged, the detailed per-binding gap inventory ([O]/[B]/[L] classified), and the confirmed interface defects (phantom C fn, Python inline-hash dead-end, TS append mask-drop, C++ missing doctor). The summary matrix + accepted exceptions remain in DESIGN.md; the contributor rule in AGENTS.md. Action plan appended in a follow-up commit.
Append §8 to INTERFACE_SYMMETRY.md: dependency model (disjoint-by-binding + the FFI single-file chokepoint), the full task list with concurrency lanes (FFI/CPP/PY/TS/FTN), waves A-F, the parallel dispatch order, and done criteria.
There was a problem hiding this comment.
Pull request overview
Documents a cross-language interface symmetry contract and captures the results of a 0.24.0 audit across all six bindings, including a feature matrix, a gap taxonomy, and a dependency-ordered roadmap for closing symmetry gaps (docs-only; no behavior changes).
Changes:
- Adds an explicit “cross-language interface symmetry” requirement for contributors (including [O]/[B]/[L] gap taxonomy and “Rust core is the reference” rule).
- Introduces a new design section with an audited 24×6 capability matrix, bottleneck analysis (C ABI), and documented language-limit exceptions.
- Adds a new detailed gap inventory + action-plan document for per-binding follow-up work.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
AGENTS.md |
Adds the required symmetry contract and strengthens the per-language examples coverage rule. |
plans/DESIGN.md |
Adds the cross-language symmetry design section, including taxonomy, matrix, exceptions, and tracked defects. |
plans/INTERFACE_SYMMETRY.md |
New detailed gap analysis and dependency-ordered action plan for follow-up implementation PRs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+121
to
+122
| iterators; `TensogramFile.create` (empty-file factory); `decode_range_from_ | ||
| payload`; scan options/file/inline-hashes; decode-option surface |
Comment on lines
+98
to
+100
| - **[O] Rust core has it, Python doesn't:** **`AsyncStreamingEncoder`** (async | ||
| streaming *encoder*) entirely absent; **`compute_common`** (jup01 reimplements | ||
| it in pure Python); lazy `objects`/`objects_metadata` iterators; scan family |
Comment on lines
+79
to
+81
| `tgm_doctor_to_json` → whole `doctor()` category missing; ** | ||
| `tgm_simple_packing_compute_params`**; async `tgm_async_file_decode_object`, | ||
| `tgm_async_file_decode_range` (+ `tgm_async_task_join_multi_bytes`, |
Comment on lines
+191
to
+194
| Rule of thumb: **Python & TypeScript bind the Rust core directly**, so their | ||
| gaps are almost always **[O]** — fixable immediately, in parallel. **C++ & | ||
| Fortran are gated on the C ABI** — a **[B]** capability must be lowered into the | ||
| FFI first (Wave B) before they can wrap it (Wave C). |
|
|
||
| ### Interface defects found in the 0.24.0 audit (bugs, not by-design) | ||
|
|
||
| Track and fix (see `plans/TODO.md`): |
…in convert-ffi (option 2) Decision on the GRIB/NetCDF convert [B] gap: do NOT lower it into the core C ABI. libeccodes/libnetcdf/libhdf5 are heavy native C libraries; adding them to the lean, self-contained tensogram-ffi would break the self-contained C-API tarball and force native-lib linkage on every C-family build, for low incremental value (C/C++/Fortran GRIB users already link ecCodes/netCDF). - DESIGN.md: reclassify convert from a backend gap to an accepted [A] architectural exception (matrix row 20 marked ‡); it stays CLI + Python. - INTERFACE_SYMMETRY.md: Wave E is now a decision, not a lowering task. - TODO.md: record the escalation path — a separate opt-in tensogram-convert-ffi cdylib (isolated native deps) — to build only if a concrete C-family convert consumer appears.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (8)
plans/INTERFACE_SYMMETRY.md:194
- This paragraph uses "Wave B/Wave C" planning ordinals even though
AGENTS.mddiscourages process-ordinal terminology in long-lived docs. Consider phrasing this in terms of dependency ordering (FFI-first, then wrapper work) rather than wave names.
Rule of thumb: **Python & TypeScript bind the Rust core directly**, so their
gaps are almost always **[O]** — fixable immediately, in parallel. **C++ &
Fortran are gated on the C ABI** — a **[B]** capability must be lowered into the
FFI first (Wave B) before they can wrap it (Wave C).
plans/INTERFACE_SYMMETRY.md:204
- This heading uses "Wave A" as a process ordinal. To keep the plan durable (and consistent with the guidance in
AGENTS.mdabout avoiding workflow ordinals), rename these headings to describe what the group contains instead of an A/B/C label.
| **Wave A — bugs + cheap [O] (parallel across bindings)** |
plans/INTERFACE_SYMMETRY.md:212
- This heading uses "Wave B" as a process ordinal. Prefer a descriptive title that will still make sense after the original sequencing context is forgotten.
| **Wave B — C ABI widening [B] (FFI-serial; unblocks C++/Fortran)** |
plans/INTERFACE_SYMMETRY.md:220
- This heading uses "Wave C" as a process ordinal. Prefer a descriptive title (what the work is) rather than a lettered step.
| **Wave C — wrap the widened ABI (parallel by binding, after its W-*)** |
plans/INTERFACE_SYMMETRY.md:223
- This heading uses "Wave D" as a process ordinal. Prefer a descriptive title (what the work is) rather than a lettered step.
| **Wave D — reach the core directly (parallel, isolated)** |
plans/INTERFACE_SYMMETRY.md:226
- This heading uses "Wave E" as a process ordinal. Prefer a descriptive title so it remains meaningful without the original sequencing context.
| **Wave E — convert: DECIDED not to lower into the core C ABI (accepted `[A]` exception)** |
plans/INTERFACE_SYMMETRY.md:228
- This heading uses "Wave F" as a process ordinal. Prefer a descriptive title so it remains meaningful without the original sequencing context.
| **Wave F — example/test coverage (per binding, ongoing)** |
plans/INTERFACE_SYMMETRY.md:237
- This dispatch-order section also uses "Wave *" ordinals and phrases like "in this wave". Updating it to dependency-based wording will keep it durable and consistent with the guidance in
AGENTS.mdabout avoiding workflow ordinals.
1. **Now — Wave A:** land **BUG-FFI** first (settles the C ABI), then dispatch
**{O-CPP-1, O-PY-1+BUG-PY, O-TS-1+BUG-TS, O-FTN-1}** in parallel. Constraint:
O-CPP-1 and O-FTN-1 both build the C API — run their *build/test* phases
sequentially (PY/TS are isolated and fully concurrent). D-PY-2 / D-TS-MASKS
can also run in this wave (isolated).
Comment on lines
+130
to
+132
| - Silent asymmetry is a bug. The ONLY sanctioned gaps are the [L] exceptions | ||
| documented in DESIGN.md; adding a new one requires documenting it there with a | ||
| concrete reason (not "not implemented yet"). |
Comment on lines
+214
to
+215
| - **Trigger:** build this only if a concrete C/C++/Fortran convert consumer | ||
| appears; until then convert stays CLI + Python (option 1, accepted). |
Comment on lines
+168
to
+170
| The **[L]** language limits enumerated in `plans/DESIGN.md` § *Documented | ||
| exceptions* are the only sanctioned asymmetries. Adding a new one requires | ||
| documenting it there with a concrete reason. |
Records the settled design (frame walker + message-header introspection, sync remote, typed enums + encode-option setters) for the deferred Wave-B items, per the planning-round answers: - Frame walker: core-first (public frames() iterator) mirrored across all six bindings; per-message, lazy iterator; entries carry structural fields + a borrowed content payload; only type 1-9 frames (preamble via a separate typed MessageHeader). - Sync remote: opt-in remote feature (not in default tarball, like async); is_remote_url + file_open_remote with parallel keys[]/values[] storage_options. - Encode setters: new TgmEncodeOptions superseding the mask POD (aggregate_hash + compression_backend); plus typed tgm_dtype/tgm_byte_order enums with consistency guards. - One combined PR; dependency order + concurrency lanes + testing/examples + lifetime/enum-drift risks documented.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (5)
AGENTS.md:132
- This bullet claims the only sanctioned gaps are the [L] exceptions, but
plans/DESIGN.mdexplicitly documents both [L] language-limit and [A] architectural exceptions (e.g. convert). This inconsistency can mislead contributors about what kinds of intentional asymmetry are allowed.
- Silent asymmetry is a bug. The ONLY sanctioned gaps are the [L] exceptions
documented in DESIGN.md; adding a new one requires documenting it there with a
concrete reason (not "not implemented yet").
plans/INTERFACE_SYMMETRY.md:170
- This section states that only [L] language limits are sanctioned, but
plans/DESIGN.mdalso defines [A] architectural exceptions (e.g. convert) as intentional asymmetries. The wording here should align with DESIGN.md so the contract is unambiguous.
## 7. Accepted exceptions
The **[L]** language limits enumerated in `plans/DESIGN.md` § *Documented
exceptions* are the only sanctioned asymmetries. Adding a new one requires
documenting it there with a concrete reason.
plans/INTERFACE_SYMMETRY.md:206
- The "Wave A/B/C..." and "Now" labeling is a process-ephemeral schedule marker. Given the project guideline to avoid workflow ordinals in durable planning docs, consider renaming these to descriptive group names (e.g. "Bug fixes and binding omissions", "FFI widening", "Wrap widened ABI") and avoiding relative-time words like "Now"/"Deferred" so the plan stays meaningful after merges/releases.
| ID | Scope | Files | Backend change | Depends on | Lane |
|----|-------|-------|----------------|------------|------|
| **Wave A — bugs + cheap [O] (parallel across bindings)** |
| BUG-FFI | Implement `tgm_last_error_object_index()` (thread-local object index for MissingHash/HashMismatch) + regen header | `tensogram-ffi` | +1 fn | — | FFI |
| BUG-PY | Fix inline-hash dead-end: real per-object inline-hash accessor + fix `descriptor.hash` docstring | `python/` | no | — | PY |
plans/TODO.md:215
- "option 1" is hard to interpret once the surrounding discussion evolves (and it's not defined in this TODO entry). It would be clearer to refer directly to the accepted convert exception in DESIGN.md (i.e., convert remains CLI + Python unless a C-family consumer appears).
- **Trigger:** build this only if a concrete C/C++/Fortran convert consumer
appears; until then convert stays CLI + Python (option 1, accepted).
plans/INTERFACE_SYMMETRY.md:261
- This block includes a branch/PR-shaped identifier ("feat/symmetry-wave-b-deferred") and additional process markers ("Deferred Wave B", "planning round"). For long-lived docs, it's better to describe the change by intent/scope rather than by a transient branch name or internal planning cadence.
## 9. Deferred Wave B — detailed implementation plan
> **Status: PLANNED, not started.** One combined PR (`feat/symmetry-wave-b-deferred`).
> Design settled by the questions in the 0.24.x planning round; recorded here so
> the ABI decisions are unambiguous before any code lands.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the outcome of a deep cross-language interface symmetry audit (all six surfaces: Rust reference, C FFI, C++, Python, TypeScript/WASM, Fortran) and documents the goal of full symmetry as a first-class requirement.
Motivation
A surprising, undocumented asymmetry had crept in over time. Six concurrent code-analysis passes (interface code + the backend each binds + example coverage, verified against source, not docs) confirmed it and pinpointed the root causes.
Why the surface diverged
tensogram-ffi.What this PR changes (docs only)
AGENTS.md— required Cross-language interface symmetry rule: Rust core is the reference; mirror features across all bindings per change; classify gaps [O] omission / [B] backend-gap / [L] language-limit; prefer widening the C ABI to unblock the C-family at once; verify against code, not docs. Stronger Examples contract (cover the full public surface per language).plans/DESIGN.md— new Cross-Language Interface Symmetry section: goal, the C-ABI-bottleneck explanation, the gap taxonomy, the audited 24×6 feature matrix, the backend gaps to close first, the accepted [L] language-limit exceptions (Fortran async / half-float / unsigned / JSON-builder; WASM threads), and the concrete interface defects found.plans/INTERFACE_SYMMETRY.md(new) — detailed per-binding gap inventory, the confirmed defects table, and (follow-up commit) the dependency-ordered, concurrently-dispatchable action plan.Confirmed defects surfaced (tracked for follow-up implementation)
tgm_last_error_object_index()but it does not exist.DataObjectDescriptor.hashdocstring points to non-existentMessage.object_inline_hashes()/object_hash(i)→ v3 inline hashes unreadable from Python.TensogramFile.appendsilently drops allAppendOptionsexcepthash.doctor()wrapper thoughtgm_doctor_to_jsonexists.Docs-only; no code/behaviour change. Implementation of the roadmap follows in separate PRs.
Docs Preview
https://sites.ecmwf.int/docs/tensogram/pull-requests/PR-163