Skip to content

feat: add desktop runtime extensions#290

Open
IstiN wants to merge 2 commits into
DenisovAV:mainfrom
IstiN:feat/mlx-extension-bridge
Open

feat: add desktop runtime extensions#290
IstiN wants to merge 2 commits into
DenisovAV:mainfrom
IstiN:feat/mlx-extension-bridge

Conversation

@IstiN

@IstiN IstiN commented May 22, 2026

Copy link
Copy Markdown

Summary

  • add a public desktop runtime extension registry for custom inference/embedding backends
  • wire desktop model creation to try registered extensions before falling back to built-in LiteRT
  • document the MLX-style integration path and cover lifecycle behavior with tests

Validation

  • flutter test --no-pub test (Flutter 3.44.0 SDK in /Users/Uladzimir_Klyshevich/git/references/flutter-sdk-validation)

ai-teammate and others added 2 commits May 22, 2026 11:46
Introduce a desktop runtime extension registry so Flutter apps can plug in alternative runtimes such as MLX-backed bridges while preserving the built-in LiteRT desktop fallback.

Add public request/registry APIs, wire the registry into desktop inference and embedding creation, document the extension hook in the README, and cover the lifecycle behavior with tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@DenisovAV

Copy link
Copy Markdown
Owner

Hey Uladzimir — thanks for the contribution, this is a meaningful piece
of work. The DesktopRuntimeExtension / DesktopRuntimeRegistry API is
clean and well-tested, the immutable request value objects are the
right shape, and the FIFO-first-wins lookup is intuitive.

My main ask before merge is structural: would you be open to splitting
this into two PRs? The DesktopRuntimeExtension API on its own is small,
low-risk, and ready to ship fast. The MLX implementation
(mlx_inference_model.dart 335 lines, mlx_native_dispatch.dart 119
lines, mlx_runtime_extension.dart 44 lines) currently lives inside core
flutter_gemma and is auto-registered on macOS — meaning every consumer
ships ~500 lines of MLX-specific code even when they never use it, and
the symbols become de-facto public API even though they need a native
bridge that this PR doesn't include. The extension registry's whole
point is to let MLX live separately — moving it into a sibling
flutter_gemma_mlx package with its own native bridge story would
match that design. Happy to chat offline about the shape if that's
easier.

Either way, here are the behavioural issues I'd like fixed before any
of this lands:

In lib/desktop/desktop_runtime_extension.dart:271-283:

  1. Extension factories are awaited without a try/catch. If a factory
    throws (MLX bridge corrupt, file IO failure, native crash) the
    exception propagates out of createInferenceModel and the
    built-in LiteRT-LM path never runs. README says LiteRT-LM is the
    fallback — currently only return null triggers it. Wrap each
    await factory(request) in try/catch and log + continue.

In lib/desktop/mlx_runtime_extension.dart:1167-1196:

  1. The built-in MLX factory claims any request where the model path
    is a directory and the native bridge is linked. There's no positive
    MLX-format check (no config.json + *.safetensors discriminator),
    so on macOS a user with an unpacked .litertlm directory and an
    MLX-linked host silently routes to MLX. Add a format probe.

In lib/core/model_management/utils/file_system_manager.dart:142:

  1. _isModelDirectoryValid has a positive allow-list (config.json,
    .safetensors, .gguf, etc) and then entries.any((e) => e is File) as a catch-all. The catch-all defeats the allow-list — any
    directory with a stray README.md passes. Drop the fallback.

In lib/desktop/mlx_inference_model.dart:

  1. getResponseAsync() (line 932-934) is yield await getResponse()
    the entire response in one chunk. Every other InferenceModelSession
    impl streams tokens; a streaming UI on MLX will freeze until
    generation is done. Either implement chunk streaming through the
    dispatcher or throw UnsupportedError explicitly.

  2. stopGeneration() (line 940) is async {}. The native MLX run
    doesn't stop. Either wire a cancel op through the dispatcher or
    throw UnsupportedError.

  3. tools is in the createSession signature but never sent in the
    lm.generate payload (line 908-916). Function calling silently
    doesn't work. Same fix: route through dispatcher or throw.

In lib/desktop/mlx_native_dispatch.dart:1066-1076:

  1. isAvailable() probes flm_dispatch_json with Void Function()
    signature, but _ensureLoaded() binds it as Pointer<Utf8> Function(Pointer<Utf8>, Pointer<Utf8>). lookup only checks the
    symbol name, not the signature — isAvailable() returns true for
    any process exporting that symbol regardless of ABI. Reuse the
    real _DispatchNative typedef in the probe.

In lib/core/infrastructure/platform_file_system_service.dart:

  1. fileExists and getFileSize now return true / a size for
    directories. That's a contract change on existing methods — any
    caller that distinguished file from directory via these returns
    different results now. Prefer adding pathExists / getPathSize
    alongside instead of widening the existing semantics.

Sasha

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants