Skip to content

Support local query embeddings with visible fallback#35

Draft
plasmaStray wants to merge 3 commits into
ClaudioDrews:mainfrom
plasmaStray:agent/local-query-embeddings
Draft

Support local query embeddings with visible fallback#35
plasmaStray wants to merge 3 commits into
ClaudioDrews:mainfrom
plasmaStray:agent/local-query-embeddings

Conversation

@plasmaStray

Copy link
Copy Markdown

What changed

  • route query-time dense embeddings through configurable EMBEDDING_API_BASE, EMBEDDING_MODEL, and optional EMBEDDING_API_KEY
  • keep query embeddings synchronous so Icarus finishes retrieval before Hermes calls the main LLM
  • add configurable timeout and retry handling for local model cold starts
  • surface degraded dense retrieval to the user through trusted pre-LLM warning context while preserving lexical/session fail-open behavior
  • retain backward-compatible OpenRouter defaults and the list-only helper API

Why

The worker already supports OpenAI-compatible local embedding endpoints, but scripts/context_enhancer.py hardcodes OpenRouter for query embeddings. That makes local indexing and query-time semantic retrieval asymmetric and silently disables dense Wiki recall for installations without an OpenRouter key.

User impact

Operators can use the same local embedding model for indexing and queries. A cold local backend is awaited before prompt construction. If dense retrieval times out or fails, the assistant is instructed to disclose that Wiki/context recall may be incomplete instead of silently answering as if semantic search succeeded.

Checks

  • python3 -m py_compile scripts/context_enhancer.py icarus/hooks.py _test_local_embeddings.py
  • python3 _test_local_embeddings.py
  • python3 _test_sanitize.py
  • python3 _test_collapse.py
  • bash -n setup.sh
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant