Enable model-selected Engine chat batching

[bmehta001]

Summary

• select Generator, static Engine batching, or dynamic Engine batching from each model's genai_config.json
• add one model-owned Engine dispatcher that keeps every Engine/Request operation on its owner thread while serving concurrent independent ChatSessions
• retain dynamic Engine requests across turns for KV-cache reuse; rebuild static Engine turns from committed history
• preserve Generator behavior for models without Engine batching and reject unsupported/ambiguous configurations
• assume ORT GenAI 0.15.3 provides the Engine API and honors configured static max_batch_size

Engine integration details

Missing assistant-turn boundary

ORT GenAI Engine does not append or retain a generated EOS token in the request's logical sequence. A later BeginTurn must therefore include the model-specific tokens that close the prior assistant turn before introducing the next user turn. The Generator path already retains EOS, so reusing its append fragment caused dynamic Engine sessions to lose conversational context even though the request and KV cache remained resident.

This change derives the continuation boundary from the model's own chat template using a synthetic assistant marker, removes the marker, and submits the remaining boundary plus new-turn fragment. This avoids hard-coding Qwen token IDs or template syntax.

Per-turn output limits

The initial Engine turn received max_output_tokens, but retained turns called BeginTurn without OgaTurnOptions. Foundry could stop reading at its external limit while the Engine still considered the turn active, making the next continuation invalid and allowing uncapped work when callers relied on the default limit.

Every Engine turn now receives an explicit resolved limit, including the 2048-token text default. Foundry waits for the corresponding terminal event, consumes exact per-turn Engine usage, and recreates retained state when request-level sampling settings that are baked into OgaGeneratorParams change.

Backend contract

Model configuration	Backend	Multi-turn behavior
engine.dynamic_batching	Shared dynamic Engine	Retain request and paged KV state
engine.static_batching	Shared static Engine	Rebuild each turn from full committed history
Neither	Existing Generator	Retain existing generator continuation
Both	Configuration error	No implicit fallback

Engine selection is limited to text chat models. Multimodal Engine declarations fail at model load rather than failing later during a request.

Validation

• Windows Release build against ORT GenAI main Engine APIs
• 49 targeted C++ configuration, search-option, template, Generator chat, concurrency, multi-turn, and cancellation tests
• Windows static Engine: single-turn, reconstructed multi-turn, cancellation, and two concurrent sessions
• Linux A100 with a CUDA paged-attention Qwen 2.5 0.5B model loaded through Foundry Local's public model-registration and CUDA-EP APIs
  • four concurrent sessions returned isolated outputs 4, 6, 8, and 10
  • retained continuation recalled sapphire while a peer request concurrently returned 14
  • per-turn usage, output limits, session destruction, model unload, unregister, and Manager shutdown completed successfully
• changed C++ files pass the repository clang-format configuration

Dependency assumption

This draft pins Microsoft.ML.OnnxRuntimeGenAI.Foundry to 0.15.3. It assumes that release contains the validated Engine C/C++ API and the static batching capacity behavior currently available in the tested upstream build.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Sep 3, 2026 11:17pm UTC