Format-metadata framework: vocabulary, formats and conversions (without wiring) by balbasty · Pull Request #310 · neuroscales/brainhops

balbasty · 2026-10-06T05:28:12Z

Closes part of #233; split out of #287, which has the review history.

Base. claude/chore/233-metadata-stack-base is a merge of #306, #308 (with #307) and #309, so the diff here shows only the metadata system. Once those merge, retarget this PR to main.

What's in it. The metadata system, used on its own. It isn't yet wired into images or transformations; that's #287.

brainhops.datamodel.metadata:
- Metadata, the common vocabulary with BIDS names, grouped into *Vocabulary classes;
- UNSUPPORTED; Scope and Along(AxisType);
- the value classes;
- ConversionReport and the loss policy;
- MetadataField;
- Metadata.load.
brainhops.io.metadata:
- FileBasedMetadata[RawT], the base of every format and the metadata dispatcher. It isn't a FileBasedObject, so io.load never returns metadata.
- OpaqueMetadata;
- the shared JSON codec;
- BIDS sidecars;
- sync_metadata.
Formats: NIfTI, MGH, Zarr, OME-Zarr, x5, ITK .h5/.tfm/.mat and FLIRT. Each one is:
- a MetadataParser (a FileParser mixin);
- two dict hooks, _decode_raw and _encode_raw;
- its public file methods.
This branch also gets the header readers it needs from the image and transformation parsers (moved here so this PR stands alone).
The format author's guide (docs/dev/metadata-formats.md) and the API pages.

Tests. Only the tests that don't need an image's .metadata field: 71 in test_datamodel_metadata.py, plus load, matrix, MGH, NIfTI and Zarr. The remaining tests come back in #287.

Stack

Design memo for the format-metadata framework #306: design memo
Add vocabulary enums for spaces, intents, manufacturers and microscopy #307: vocabulary enums
Add core helpers: shortest_decimal, EnumConverter, own_annotations #308: core helpers
Extract the FormatDispatcher mixin; accept non-tuple image indices #309: FormatDispatcher mixin and small fixes
this PR: the metadata system
Wire format metadata into images and transformations (#233) #287: wiring, stacked on this PR

🤖 Generated with Claude Code

https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

Generated by Claude Code

brainhops.datamodel.metadata: Metadata, the format-agnostic metadata of an image or a transformation, declared as six vocabulary groups (provenance, acquisition, diffusion, display, microscopy, storage) with their BIDS keys, scopes and per-axis fields; the UNSUPPORTED sentinel and supports= capabilities; the raw record and its read-time snapshot; conversion between formats (Metadata.to) with ConversionReport and the loss policy; derive, _select and _reslice for derived images; the value classes (EncodingDirection, GeneratedBy, Channel); preferred_dtype and preferred_storage; Metadata.load, from_bids and to_bids (which import brainhops.io lazily); and MetadataField, the annotation that the metadata field of an image or a transformation will use. bagof-magic >= 0.3.dev2 is required. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

…ta (#233) - io/metadata: FileBasedMetadata, the dispatcher of the formats whose files hold metadata (Metadata.load), OpaqueMetadata, the JSON codec of key/value stores, the BIDS sidecar, and sync_metadata, the helper a parser will call to keep its metadata in step with its raw record. - io/base/_metadata_parser.py: MetadataParser and Hdf5MetadataParser. - The metadata of each format, registered with FileBasedMetadata and exported next to its image or transformation classes: NiftiMetadata, MghMetadata (with the MGH tags codec), ZarrMetadata and OmeZarrMetadata, X5Metadata, ItkMetadata and ItkH5Metadata, and FlirtMetadata. - The readers they share with the image and transformation parsers: _load_nifti_header reads a stream too (and the header as stored), is_nifti_stream, _geometry_time_step; read_mgh_raw, is_mgh_stream; the Zarr node_attributes and write_attributes; read_h5_header, with H5Header moved to itk/_metadata.py. No image or transformation has a metadata field yet: the tests cover the metadata classes, their codecs, Metadata.load and the conversion matrix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

docs/dev/metadata-formats.md explains how to write the metadata of a format; the API pages document brainhops.datamodel.metadata (and the names for format authors), brainhops.io.metadata and the ITK metadata. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

balbasty · 2026-10-06T08:14:45Z

+        # `UNSUPPORTED`, `None` and an empty `extra` are hidden, or a
+        # format that stores three fields would print forty. `format` and
+        # a format's own fields come first, then `extra`, then the
+        # vocabulary in its declared order (not `bagof`'s reverse MRO).


Look at the latest dev release of bagof magic, which allows specifying more complex repr policies.

Done in 3c4238e. The custom __repr__ is gone. bagof-magic 0.3.dev2 has HideIf, so:

a class-level repr=HideIf(_is_absent) on Metadata and Vocabulary hides None and UNSUPPORTED;

extra has HideIf(operator.not_), so it is hidden when empty.

With reverse=True, generic metadata prints format, then extra, then the vocabulary.

One gap remains: a format class prints format last, e.g. NiftiMetadata(description='x', format='nifti'). The repr follows field order, and the on= pin leaves format where the parent declared it. Fixing that would need a per-field repr position option upstream. Is format-last acceptable, or should I ask bagof for one?

Generated by Claude Code

balbasty · 2026-10-06T08:20:46Z

+    # --- files ------------------------------------------------------
+
+    @classmethod
+    def load(cls, file: tx.Any, **kwargs: tx.Any) -> "Metadata":


This should only be in the io version of metadata (filebased). No io ops on the base class.

Removed in 510a2dd. A file's metadata is read with brainhops.io.metadata.FileBasedMetadata.load, which is the dispatcher; its docstring now lists the formats and gives examples. Tests, the guides and the memo are updated, and so is #287.

Generated by Claude Code

balbasty · 2026-10-06T08:21:22Z

+    # --- BIDS ---------------------------------------------------------
+
+    @classmethod
+    def from_bids(cls, sidecar: tx.Any) -> "Metadata":


Removed in 510a2dd. Sidecars are read with brainhops.io.metadata.bids.from_bids, or the BidsSidecar format.

Generated by Claude Code

balbasty · 2026-10-06T08:21:35Z

+
+        return from_bids(sidecar)
+
+    def to_bids(


Removed in 510a2dd. Sidecars are written with brainhops.io.metadata.bids.to_bids(meta).

Generated by Claude Code

balbasty · 2026-10-06T08:39:01Z

+# ----------------------------------------------------------------------
+
+
+def _is_absent(value: tx.Any) -> bool:


In general, and when possible, move private helpers below public helpers which are below public classes or main functions.

Done in 170af7d, for every module the PR adds: public code first, then public helpers, then private helpers. Seven private definitions must exist first at import time (_rgba, _vector, _space, _History, _T, _NEW, and the _ZarrMetadataParser base), and each has a one-line comment saying why.

Generated by Claude Code

balbasty · 2026-10-06T09:00:20Z

+        """
+        return cls.from_fileobj(BytesIO(content), **kwargs)
+
+    def to_file(self, file: path.FileLike, **kwargs: tx.Any) -> None:


There is no to_ on (non writeable) FileParsers anyway.

Removed in 2f9493d. Only plain Zarr metadata defines to_file. The test now asserts that NIfTI, MGH and OME-Zarr metadata have none.

Generated by Claude Code

balbasty · 2026-10-06T09:00:53Z

+    _READ_MODE = "rb"
+
+    @classmethod
+    def from_bytes(cls, content: bytes, **kwargs: tx.Any) -> tx.Any:


Why not have this on the FileParser? Is it better than the current one?

Kept on purpose, with a docstring in 2f9493d. The reason is in how FileParser reads:

FileParser.from_bytes raises ParserNotImplementedError;

FileParser.from_fileobj reads the whole stream and hands it to from_bytes.

Metadata formats implement only from_fileobj, since they read just a header. Without the override, FileBasedMetadata.load(raw_bytes) would fail; test_io_metadata_load covers it.

As a FileParser default it isn't strictly better. The two defaults would then call each other, so a parser that overrides neither would recurse forever. That needs a guard, e.g. only when the class overrides from_fileobj. I've left it for #297.

Generated by Claude Code

balbasty · 2026-10-06T09:01:43Z

+        )
+
+
+class Hdf5MetadataParser(MetadataParser):


Why do we have format specific sniffers in the general io metadata module?

Moved in 2f9493d:

Hdf5MetadataParser now lives next to Hdf5Parser in io/base/hdf5.py, and derives from it. That drops about 150 lines of duplicated path/stream/bytes plumbing.

Since that module needs h5py, ItkH5Metadata and read_h5_header moved to itk/h5/_metadata.py. itk exports ItkH5Metadata only when h5py is installed.

Generated by Claude Code

balbasty · 2026-10-06T09:37:11Z

+        float
+            The confidence, in `[0, 1]`.
+        """
+        from brainhops.io.base.nifti import is_nifti_stream


Why not top import ?

In 30fa2dc, imports moved to the top wherever no cycle exists. For example, the Zarr metadata now imports _ome and abczarr.open there; node_attributes/write_attributes moved into _metadata to break a cycle with _image.

This one has to stay lazy, now with a comment: nifti.py imports _nifti_metadata, which imports from nifti.py. MGH is in the same situation.

Generated by Claude Code

The groups (Vocabulary and its seven subclasses) are what a format names in supports=, so they are public names of brainhops.datamodel.metadata, and the docs that list the public names say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

…icies (#233) The custom __repr__ goes: Metadata and the vocabulary groups hide a field that holds None or UNSUPPORTED (repr=HideIf(...), bagof-magic 0.3.dev2, the current floor), and extra is hidden when empty too. The fields are listed in reverse, with the groups in reverse order, so that generic metadata shows format, extra, then the vocabulary in its declared order. A format class declares the vocabulary again, and bagof cannot place format before it, so a format's repr shows format last. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

Metadata.load, Metadata.from_bids and Metadata.to_bids go: the data model reads and writes no file. The metadata of a file is read by FileBasedMetadata.load (the dispatcher, or the class of a format), and a BIDS sidecar by brainhops.io.metadata.bids (from_bids, to_bids, and the BidsSidecar reader). The tests, the format author's guide and the design memo follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

…unctions (#233) No metadata class overrides Metadata._convert_from, so it is the private function _convert_from(cls, other, args, kwargs). The two _accepts_raw methods and FileBasedMetadata._raw_type give way to the class variable _raw_class, read by the function _accepts_raw(cls, raw): Metadata declares None (any record) and FileBasedMetadata type(None) (none) until a format declares its own type, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

…rmation (#233) The space of an EncodingDirection may be a brainhops CoordinateSystem, kept as it is and equal only to an equal system. JSON names a space with a string, so the codec writes a system by its name, and encode_changes (which now takes the report) reports as lost a direction in a system without a name. EncodingDirection.transform also takes a Transformation, through the linear part of the affine it reduces to (_as_affine), and raises TypeError for one that does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

stored_values(data, dtype, slope, intercept) is what a writer stores with the type and the scaling preferred_storage chose: the unscaled values, rounded only when dtype is an integer type. The docstring of preferred_storage says so, and that a scaling is only returned with an integer type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

…Parser (#233) MetadataParser loses to_file, which only refused: a FileParser does not write, and plain Zarr defines its own. It keeps from_bytes, documented: a format implements from_fileobj, and FileParser routes bytes the other way (a default for binary parsers is issue #297). Hdf5MetadataParser moves next to Hdf5Parser (io/base/hdf5.py) and derives from it, whose routing of paths, streams and bytes it used to duplicate; sniff_h5 takes error= as for Hdf5Parser. ItkH5Metadata and read_h5_header move next to the ITK .h5 parser (itk/h5/_metadata.py), since they need h5py, and itk exports ItkH5Metadata when h5py is installed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

The Zarr metadata imports _ome and abczarr.open at the top, and the attribute helpers (node_attributes, write_attributes) move from _image to _metadata, which the image imports. The imports that stay in a function cross a real cycle, and say so: _nifti_metadata and _mgh_metadata import from nifti and mgh, which import them, EncodingDirection.transform imports the transformations, whose metadata field imports the metadata, and _with_brainhops reads brainhops.__version__. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

… helpers (#233) Every module the metadata adds lists its public classes and functions first, then its public helpers, then its private ones. A private definition that must exist first at import time (a converter a field annotation evaluates, a base class, a default, a TypeVar) stays above, with a comment that says why. No behaviour changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

…tem) into the wiring Brings in the answers to the review of #310: the public vocabulary groups, the repr built by bagof, no input or output on Metadata, the conversion helpers as functions, directions in a CoordinateSystem, the rounding of scaled values, the read-only metadata parser and the HDF5 one next to Hdf5Parser, and the layout of the modules. The ITK .h5 parser takes ItkH5Metadata and read_h5_header from itk/h5/_metadata. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

…ages (#233) `Metadata.derive(operation)` replaces the private hooks `_select` and `_reslice`: an image operation describes what it did as an `Operation` (`Indexed` for an index, `Resampled` for a resampling), and each value propagates through it by the handler registered for its type, or else for the scope of its field (`propagates`, `propagate`, in the new module `_operations`). The encoding direction registers its handler in `_terms`, the scope defaults are registered under `Scope`, and the raw record of a format goes through `propagate_raw`, which NIfTI uses to scrub its header in place of its `_reslice` override. The index expansion and the axis types move from the images into `Indexed`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

#233) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

…233) The class name already says the format. `FileBasedMetadata` declares `format` again with `HideIfDefault`, which bagof binds again on each class against the value the class pins, so `NiftiMetadata(...)` hides `format='nifti'`. Generic `Metadata` keeps showing its format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

…rser (#233) Since #311, `FileParser.from_bytes` wraps the bytes in a stream for a class that implements `from_fileobj`, which every metadata format does, so `FileBasedMetadata.load(raw_bytes)` needs no override. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

A level of a pyramid is derived when the pyramid is opened, and its geometry needs its shape, so its data; the level passes none, and no handler reads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

balbasty · 2026-10-06T12:18:15Z

Operations-based derive (41ceecc, 2390b07 here; 7fce768 on #287). This replaces _select/_reslice, following the design discussed in the session.

Operations: Indexed(index, shape, system) and Resampled(transformation, geometry=None) (datamodel/metadata/_operations.py), passed to derive(operation, *, history=...). The provenance rules are unchanged.
Handlers: one registry, @propagates(key, OperationType), where key is a value class or a Scope.
- EncodingDirection registers its own handler, in _terms.
- The scope defaults sit under Scope and reproduce today's results.
- An operation type nothing handles clears the spatial fields and keeps the rest.
Raw records: handled through the same registry. NIfTI registers a handler for Nifti1Header in place of its _reslice override.
Removed: _select, _reslice, _map_direction, _index and the changed= convention. On Wire format metadata into images and transformations (#233) #287, the image helpers _derived_metadata, _expand_index, _axis_types and _linear_part are gone too.
img(transform): stays a plain copy, with no provenance entry.

Also here

MetadataParser.from_bytes is removed (f1c1c56); FileParser.from_bytes falls back to from_fileobj #311's FileParser default covers it. main is merged into the stack base and into this branch.
Format classes hide format in their repr, through HideIfDefault on FileBasedMetadata (2b6e1cb).

Generated by Claude Code

claude added 3 commits October 6, 2026 05:21

balbasty mentioned this pull request Oct 6, 2026

Wire format metadata into images and transformations (#233) #287

Draft

balbasty commented Oct 6, 2026

View reviewed changes

claude added 9 commits October 6, 2026 09:48

balbasty mentioned this pull request Oct 6, 2026

FileParser.from_bytes falls back to from_fileobj #311

Merged

claude added 6 commits October 6, 2026 11:25

Docs(metadata): the operation objects in the format guide and the memo (

8c3e387

#233) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

Merge the stack base (origin/main with #311) into the metadata system

0f25da0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmK1jLTwd2hMkvYnMWEyYw

This was referenced Oct 9, 2026

Refactor(io): simplify format dispatch and reader/writer APIs #357

Merged

Format classes should hold their parsed record, not inherit their parser #415

Open

		# ----------------------------------------------------------------------


		def _is_absent(value: tx.Any) -> bool:

Conversation

balbasty commented Oct 6, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

balbasty commented Oct 6, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants