[RNE Rewrite] refactor!: add better model schema contract and validation logic

[barhanc]

Description

Intoduces a new powerful and extensible mechanism for validating model contracts both during the runtime execution (mitigating e.g. internal error crashes when two input tensors should have the same dimension during runtime) and pipeline construction (matching model contract against a specified interface).

• Introduces consistent, clean, simple and extensible contract for custom schema specification as an optional constant companion method get_model_schema returning string-encoded JSON. No need for constructing auxiliary tensors for specifying dynamic dimensions, multiple companions for different kinds, etc.
• Introduces distinction between symbolic dims and concrete dims which represent dimension domains the model can require (includes constant, enumerated and range dims).
• Adds extensible mechanism of specifying runtime constraints (such as dynamic dim equality, linearity) which is distinct from dim domain constraints.
• Adds ergonomic helper functions which make the model spec specification in pipeline implementation concise yet clear.

Introduces a breaking change?

• Yes
• No

Type of change

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Documentation update (improves or adds clarity to existing documentation)
• Other (chores, tests, code style improvements etc.)

Tested on

• iOS (Simulator only, re-exported MLX and CoreML haven't been tested)
• Android

Testing instructions

Models that use the companion method to get the JSON schema have not yet been updated on the SWM HF as they wait for the review of this PR. For testing you can use the models at https://huggingface.co/bhanc/scratch/tree/main/ptes.

• Build all example apps on both Android and iOS and check that all models still work

Screenshots

Related issues

Closes #1323

Checklist

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings

Additional notes

• The updated export scripts are on the internal gitlab on the @bh/schema-workflow branch.
• @msluszniak I haven't been able to re-export the LFM2.5 embeddings from the @nk/universal-exporter branch. I would be grateful if you could take a look.
• @benITo47 Please take a look if you can find some use-case which this mechanism cannot express.

[msluszniak]

@barhanc I updated the export script so know it should work correctly (at least I can export them my myself from Norbert's branch). The problem was in the newest version of transformers. Please give it a try.

[msluszniak]

For text embeddinga, I got two errors:

The rest of the text embedding models worked correctly.

[msluszniak]

Regarding vad and whisper, I only had problems with coreml variants of whisper. For all that I was able to run on Iphone SE, I get the following error:

[barhanc]

The models should be fixed now. I've updated the .pte files on the https://huggingface.co/bhanc/scratch/tree/main repo for now. After you test that everything is fine I will upload them to the SWM HF under v0.10.0 tag.

[msluszniak]

Now everything worked correctly, @benITo47 are you fine with these changes?

[benITo47]

@msluszniak

From what I've been reading, changes seem sane. One real gap for OCR: enumerated shapes are a per-dimension cross product. tensor_helpers.cpp fromJs validates each dim against its own EnumDim.choices independently, and dims.enum only exposes per-dim arrays. There's no whole-shape domain, and runtimeConstraints (eq/linear) are dim-relations, so they can't carve out a sparse set either.

For example, our CoreML detectors ship sets like {640², 960², 1280², 1280×640} for PaddleOCR. To describe that in the new schema I can only give H:{640,960,1280}, W:{640,960,1280} - whose cross product also admits 640×1280, which the PTE never compiled. fromJs green-lights it, then CoreML rejects it at execute.

While resizing to match input, im picking which shape to use (snap up or snap down), in that process I can actively select that illegal shape.

To make the per-dim schema match the compiled model I'd have to either compile the full cross product (more ANE specializations, more coreML overhead, worse performance) or drop to square-only (lose the portrait bucket). The workaround is to hardcode the whole-shape list in the task and snap to it - but then the model doesn't self-describe its contract, code breaks when hardcoded list drifts from the export, and inspectModel shows a wrong over-permissive shape set. So at minimum this should be expressible in schema.ts - a whole-shape / tuple-enum domain (or constraint kind), whatever fits the model best.

[barhanc]

@msluszniak I've rebased, updated the models on SWM HF and tested them again (the ones I could on a simulator). If there is a green light I will merge this PR.

[msluszniak]

There is a green light indeed, today I tested both coreml and mlx variants of uploaded files.