Integrate dynamic batching and in-memory audio cache features

[hdduytran]

merge: integrate dynamic batching + in-memory audio cache

What this does

Merges two independently developed feature branches into main:

1. feature/dynamic-batching — groups concurrent requests into single batched model.generate() calls
2. feature/in-memory-cache — LRU cache with TTL for repeated synthesis requests

Both branches were based off main independently (no stacking). This integration branch merges them sequentially, resolves conflicts, and verifies the combined result.

Why a merge branch

The two features touch overlapping files (config.py, app.py, health.py, README.md, docs/architecture/overview.md, docs/design/dataflow.md) but are functionally independent. Merging via an integration branch lets us resolve all 6 conflicts in one place and verify the combined behavior before landing on main.

Conflicts resolved

File	Conflict	Resolution
config.py	Both added fields after request_timeout_s	Combined: batching fields first, then cache fields
app.py	Both added imports and shutdown logic	Combined: import both services, shutdown both in sequence
routers/health.py	Both added fields to /health response	Combined: batch config + cache config in same response
README.md	Both added feature bullet and Advanced Features section	Combined: both bullets, both sections (batching then cache)
docs/architecture/overview.md	Both modified service layer diagram, component map, dependency graph, startup/shutdown sequence	Combined: BatchingService + AudioCache in all diagrams
docs/design/dataflow.md	Both modified /health and /metrics JSON examples	Combined: all fields from both features in responses

How the features interact

They don't — by design. The request flow is:

Request → Cache lookup → HIT? return cached bytes
                       → MISS? → BatchingService (or legacy single-request)
                                → model.generate()
                                → encode audio bytes
                                → Cache store
                                → return to client

Cache sits at the HTTP layer (keyed on request params, stores final bytes). Batching sits at the inference layer (groups model.generate() calls). They compose naturally with no coupling.

Merge history

main (ec8b4a4)
├── feature/dynamic-batching (2 commits)
│   ├── feat: add dynamic batching for inference
│   └── docs: add dynamic batching documentation
└── feature/in-memory-cache (2 commits)
    ├── feat: add in-memory LRU cache for repeated synthesis requests
    └── docs: add audio cache documentation

integrate/batching-and-cache
├── merge: integrate dynamic batching feature (no conflicts)
└── merge: integrate in-memory audio cache feature (6 conflicts resolved)

Verification

• 0 conflict markers remaining (grep -rn "<<<<<<" — clean)
• 0 diagnostics issues across all 7 modified source files
• 40/40 tests passing

[maemreyo]

The cache-before-batching ordering is exactly right! Short-circuiting early for cache hits avoids all the batching overhead. Have you considered adding metrics to track how often cache hits prevent batch formation? It could be interesting data for tuning the batch timeout!

[maemreyo]

The startup/shutdown sequence is well thought out. I like that batching service starts before cache - ensures inference pipeline is ready before caching layer. One minor thought: should the executor shutdown happen after cache cleanup? If cache cleanup triggers any async operations that need the executor, we might have issues. Though looking at the code, cache stop just cancels the sweep task so it's probably fine.

[maemreyo]

Having both features enabled by default is the right call for production deployments. One question: have you tested the interaction when both are enabled? Specifically:

1. Cache hit rate impact on batch formation (fewer requests reach batching)
2. Memory pressure with both caches (audio cache + batch queue)

[maemreyo]

With both features enabled, a "cache prevented batches" metric would be useful - showing how many potential batch slots were saved by cache hits. Helps users understand the interaction between the two optimizations.

[maemreyo]

With both features enabled, a "cache prevented batches" metric would be useful - showing how many potential batch slots were saved by cache hits. Helps users understand the interaction between the two optimizations.

[maemreyo]

The cache key coverage looks comprehensive. One edge case: if two requests have identical everything except response_format (WAV vs PCM), they'll be cached separately. This is correct behavior but worth noting in docs since the same inference produces different cached bytes

[maemreyo]

That's 6 new CLI flags - quite a lot! Have you considered grouping them?

# Current
omnivoice-server --batch-enabled --batch-max-size 16 --cache-max-mb 2048

Alternative: config profiles
omnivoice-server --performance-profile high-throughput

Not necessary for this PR, but food for thought for future UX improvements. The current explicit flags are clearer for power users.

[maemreyo]

The sequence diagrams showing cache lookup → batch accumulation → GPU dispatch are excellent! This makes the request lifecycle much clearer for new contributors. One suggestion: add a "cache hit" path diagram showing the short-circuit flow for completeness.

[maemreyo]

Hi @hdduytran Im adding some feedback when walking through your excellent work. Please feel free to reach out, we can discuss more bout those.
Once again, thanks for your dedicated work on this!!