[RNE Rewrite] feat!: add classification task and common utilities for manipulating tensors in CV tasks by barhanc · Pull Request #1264 · software-mansion/react-native-executorch

barhanc · 2026-06-18T12:07:24Z

Description

This PR adds:

common utilities such as useModel and useResourceDownload hook which are needed for all models
common utilities and scaffolding for computer vision tasks (including native operations for image tensors manipulation)
scaffolding for computer vision example app
classification task with testing screen in the example app
inspection utilities and screen in the example app to facilitate easier exports in the future for other tasks

It also introduces some fixes:

flexible pages support on Android
iOS bridge initialization for the RN new architecture
configures prettierrc to 100 column width
adds documentation to core functionalities (both on TS and native side)

Introduces a breaking change?

Yes
No

Type of change

Bug fix (change which fixes an issue)
New feature (change which adds functionality)
Documentation update (improves or adds clarity to existing documentation)
Other (chores, tests, code style improvements etc.)

Tested on

iOS
Android

Testing instructions

Build the example computer-vision app and confirm everything builds correctly on both iOS and Android.
Run the classification task and validate the results.
Check the inspection screen (try to inspect some models from our HF).

Screenshots

N/A

Related issues

Closes #1237
Closes #1207

Checklist

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly
My changes generate no new warnings

Additional notes

…extension sources in build files

…age preprocessing

…e classification task

…ourceDownload, and register public exports

…amelCase in ESLint

…eaders

Align native extensions with the add-native-extension guideline: - sigmoid: use the precomputed src->numel_ instead of recomputing the element count via std::accumulate; drop the now-unused <numeric>. - softmax/argmax: require axis (count == 3) instead of defaulting to -1 in C++; defaults belong in the TS wrapper layer. argmax now validates args[2].isNumber() (parity with softmax) and registers with arity 3. - normalize: drop misleading "default" init for the required alpha/beta coefficients. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Download to a .partial temp file and promote it to the final cache path only after the transfer succeeds. Previously an interrupted download left a truncated file that a later run treated as a valid cache hit, loading a corrupt .pte. Also treat HTTP >= 400 as a failure (RNFS.downloadFile resolves on 4xx/5xx) and unlink the partial file on any error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Surface the resize pad value through the preprocessor options so letterbox consumers can configure it instead of being stuck with black padding. Optional; the resize wrapper defaults it to 0. Switch the resize wrapper to per-key nullish defaults so an explicit `padValue: undefined` falls back to the default instead of clobbering it through object spread (the native op requires padValue unconditionally). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…n screen

…erties

msluszniak

Maybe we change example app icon to something meaningful for CV, I thought about it for our current apps, but haven't executed eventually.

…anonymous namespaces

…::fill, and clean up loop/dimension types

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

…ttier-ignore

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

…in error message

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

Each useResourceDownload instance now writes to a unique .partial file so concurrent downloads of the same source no longer clobber a shared temp file. The moveFile call is guarded so a download that loses the race to an identical file already at the destination (iOS moveFile throws when dest exists) is treated as success instead of surfacing a spurious error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

… platforms

barhanc · 2026-06-19T14:03:04Z

Regarding example apps visual aspects, we can open an issue regarding the general overhaul of example apps, but I don't think it is a priority right now.

msluszniak · 2026-06-19T14:05:57Z

Regarding example apps visual aspects, we can open an issue regarding the general overhaul of example apps, but I don't think it is a priority right now.

Agreed. Let me do another quick pass over the code ;)

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

## Description Adds the tokenizer pipeline (issue #1248) using the new worklet-based architecture, with functional parity to the current `TokenizerModule`. A new `nlp` extension exposes a `loadTokenizer` JSI primitive (top-level on `__rnexecutorch_jsi__`, like `loadModel`) returning a `Tokenizer` host object backed by `tokenizers::HFTokenizer`. On top of it sits a `createTokenizer(config, runtime?)` async factory (async + `*Worklet` variants + `dispose`) and a `useTokenizer` hook. Methods: `encode`, `decode`, `getVocabSize`, `idToToken`, `tokenToId` — same semantics as today (special tokens follow the `tokenizer.json` post_processor). The `*Worklet` variants let an upcoming text-embeddings task tokenize → build tensors → run forward within a single worklet. - C++: `cpp/extensions/nlp/{tokenizer,install}.{h,cpp}`, wired into `RnExecutorch.cpp`. - TS: `src/extensions/nlp/{ops,tasks}/tokenizer.ts`, `src/hooks/useTokenizer.ts`, exports in `index.ts`, example `models.tokenizer.ALL_MINILM_L6_V2`. - Build: tokenizer header-search paths added to `android/CMakeLists.txt` and the podspec — `pytorch/tokenizers/include` plus the bundled libs its public headers pull in (`nlohmann/json`, `re2`, and re2's `abseil` dep). Symbols link from the prebuilt `libexecutorch`. Documented in `third-party/README.md`. - Demo: a dedicated `apps/nlp` example app with a Tokenizer screen that drives the full pipeline on device. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [x] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [ ] Android ### Testing instructions CI is TypeScript-only here (native isn't compiled in CI); `yarn typecheck`, root `yarn lint`, and `yarn prepare` (bob build) all pass. To exercise the native tokenizer end-to-end via the demo app: 1. Provision the ExecuTorch `third-party` artifacts into `packages/react-native-executorch/third-party/` (see `third-party/README.md`). The existing PoC bundle already ships the llm/tokenizers extension (headers + symbols), so no rebuild is needed. 2. `yarn && cd apps/nlp && yarn ios` — the iOS simulator works since the tokenizer is pure CPU (no GPU/Metal). 3. Open the **Tokenizer** screen. It loads `all-MiniLM-L6-v2` and auto-runs `encode` / `decode` / `getVocabSize` / `idToToken` / `tokenToId`, asserting: `encode("Hello world")` = `[7592, 2088]`, `decode` round-trips to `"hello world"`, `getVocabSize()` = `30522`, and `tokenToId(idToToken(id))` is the identity. All three assertions should read **PASS** (also logged as `[TokenizerTest]`). The screen genuinely drives the new code — `useTokenizer` → `createTokenizer` → the `loadTokenizer` JSI primitive → the native `TokenizerHostObject` / `HFTokenizer` — so a green run validates the whole pipeline, not just the types. Verified locally on iOS (iPhone 16 Pro Max, Xcode 26.5): all assertions pass. The `apps/nlp` app is intentionally minimal and exists only to prove this pipeline; it can be dropped after approval if you'd prefer not to keep a demo app in-tree. ### Screenshots ### Related issues #1248, part of #1208 ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes The C++ mirrors the current `TokenizerModule`, which is backed by `pytorch/tokenizers` (`tokenizers::HFTokenizer`) from the ExecuTorch llm/tokenizers extension. This PR consumes the same headers (the third-party bundle under `extension/llm/tokenizers`) and prebuilt symbols (from `libexecutorch`); it does not use the `tokenizers-cpp` submodule. Tokenizer download currently uses the temporary `react-native-fs`-based `useResourceDownload` introduced in #1264 (to be replaced by the ResourceFetcher in #1253).

## Description - Adds semantic segmentation task and required native operations for it. - Adds computer-vision example app screen for semantic segmentation. - Fixes the performance regression in the argmax introduced in #1264. The initial implementation from PoC was more efficient for the default `axis=-1` case as the internal loop was over contiguous elements. The linked PR changed it so that it was more efficient for `axis=0` case, however since the default is `axis=-1` this caused a performance regression in the semantic segmentation task. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [x] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [x] Android ### Testing instructions - [ ] Build the computer-vision example app. - [ ] Test the semantic segmentation screen.  ### Screenshots  ### Related issues  Closes #1242 ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes  --------- Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

barhanc added 7 commits June 18, 2026 13:10

chore: configure prettier printWidth and update spellcheck wordlist

f6992f2

build: add react-native-fs dependency, configure OpenCV, and include …

486d8d3

…extension sources in build files

feat(cpp): implement native JSI extensions for math operations and im…

1d0c88e

…age preprocessing

feat(ts): implement math/cv extensions, image preprocessing, and imag…

a10cb70

…e classification task

feat(hooks): implement classifier hooks, inspectModel utility, useRes…

c86b86e

…ourceDownload, and register public exports

docs: add JSDoc comments to hooks, tasks, and utils; enforce strict c…

c8d6a14

…amelCase in ESLint

docs(native): add Doxygen comments to native JSI core and extension h…

3380bba

…eaders

barhanc self-assigned this Jun 18, 2026

barhanc added the feature PRs that implement a new feature label Jun 18, 2026

barhanc changed the title ~~feat: add classification task and common utilities for manipulating tensors in CV tasks~~ [RNE Rewrite] feat: add classification task and common utilities for manipulating tensors in CV tasks Jun 18, 2026

barhanc and others added 8 commits June 18, 2026 15:27

fix: fix native iOS bridge for new architecture

231eb4d

docs: add JSDoc to core tensor, model, modelSchema, and runtime

9d09d0e

fix: misc small fixes

b6ad29a

fix: misc small fixes

b6240c7

feat(app): add computer-vision example app scaffold and classificatio…

c99eed6

…n screen

barhanc changed the title ~~[RNE Rewrite] feat: add classification task and common utilities for manipulating tensors in CV tasks~~ [RNE Rewrite] feat!: add classification task and common utilities for manipulating tensors in CV tasks Jun 18, 2026

barhanc marked this pull request as ready for review June 18, 2026 16:34

barhanc requested a review from msluszniak June 18, 2026 16:34

This was linked to issues Jun 18, 2026

[RNE Rewrite] CV - add classification pipeline implementation #1237

Closed

Add utilities for inspecting .pte files directly #1207

Closed

barhanc added the refactoring label Jun 18, 2026

docs(core): add documentation for Tensor dtype, shape, and numel prop…

d744091

…erties

msluszniak reviewed Jun 19, 2026

View reviewed changes

barhanc added 3 commits June 19, 2026 13:41

style: add braces around if statements and migrate static helpers to …

1b08fbd

…anonymous namespaces

style: rename local C++ variables to camelCase

ff47724

refactor(cv): de-duplicate normalize options parsing, use std::ranges…

cd90139

…::fill, and clean up loop/dimension types

barhanc and others added 11 commits June 19, 2026 14:39

Apply suggestion from @msluszniak

d4c3088

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

fix: type

52f8ccb

Assert label vocab length at construction, validate topk, and add pre…

5d7f18a

…ttier-ignore

Apply suggestion from @msluszniak

daecf86

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

refact(cv): use constant for RGB channels and format expected shapes …

2c161ff

…in error message

Apply suggestion from @msluszniak

4de3987

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

fix: dispose loaded model in finally block in inspectModel

2cacd32

refactor(cv): use computeFit helper in resize operation

af7c860

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

refactor(math): use size_t in loops for axis operations

d8d2fb6

feat(computer-vision): conditionally disable CoreML option on non-iOS…

4cb66b0

… platforms

barhanc requested a review from msluszniak June 19, 2026 14:01

msluszniak reviewed Jun 19, 2026

View reviewed changes

Comment thread packages/react-native-executorch/cpp/extensions/math/operations.cpp Outdated

msluszniak reviewed Jun 19, 2026

View reviewed changes

Comment thread packages/react-native-executorch/cpp/extensions/cv/image_ops.cpp Outdated

msluszniak reviewed Jun 19, 2026

View reviewed changes

Comment thread packages/react-native-executorch/src/utils.ts Outdated

msluszniak reviewed Jun 19, 2026

View reviewed changes

Comment thread apps/computer-vision/components/ModelPicker.tsx Outdated

barhanc and others added 2 commits June 19, 2026 16:27

Apply suggestions from code review

2a8a73f

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>

refactor(cpp): use std::cmp_less for signed/unsigned loop comparisons

84e309d

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

barhanc requested a review from msluszniak June 19, 2026 14:40

msluszniak approved these changes Jun 19, 2026

View reviewed changes

barhanc merged commit 6a7483d into rne-rewrite Jun 19, 2026
2 checks passed

barhanc deleted the @bh/classification-task branch June 19, 2026 15:03

This was referenced Jun 19, 2026

[RNE Rewrite] CV - add classification pipeline implementation #1237

Closed

Add utilities for inspecting .pte files directly #1207

Closed

msluszniak mentioned this pull request Jun 22, 2026

[RNE Rewrite] feat: add tokenizer pipeline (#1248) #1274

Merged

12 tasks

barhanc mentioned this pull request Jun 22, 2026

[RNE Rewrite] feat: add semantic segmentation task #1275

Merged

14 tasks

Uh oh!

Conversation

barhanc commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Introduces a breaking change?

Type of change

Tested on

Testing instructions

Screenshots

Related issues

Checklist

Additional notes

Uh oh!

msluszniak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

barhanc commented Jun 19, 2026

Uh oh!

msluszniak commented Jun 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

barhanc commented Jun 18, 2026 •

edited

Loading