feat: Multi-Modal Embedding Fusion

[aGallea]

Problem

Image and text embeddings live in separate ChromaDB collections with different vector spaces. Users can index imageUrl and productDisplayName separately, but there is no way to combine them into a single unified embedding that captures both visual and semantic information.

For example, two products might look similar (same color, shape) but have very different descriptions, or vice versa. Separate collections miss these cross-modal relationships.

Proposed Solution

Add a fusion mode during indexing that creates a unified multi-modal vector from image and text embeddings. Three fusion strategies:

1. Concatenate — Join image and text vectors end-to-end (512 + 384 = 896 dims). Simple, preserves all information, but higher dimensionality.
2. Weighted Average — Average the vectors with configurable weights (requires same dimensionality — use CLIP for both modalities).
3. CLIP Unified — Use CLIP's text encoder alongside its image encoder so both modalities live in the same 512-dim latent space, then average.

Implementation Details

Backend

Modify: embedding_cluster/settings.py

• Add fusion_mode: str | None = Field(default=None, description="Embedding fusion mode: none, concatenate, average, clip_unified")
• Add fusion_weights: list[float] | None = Field(default=None, description="Weights for weighted average fusion, e.g. [0.7, 0.3] for image-heavy")
• Validation: fusion_weights must sum to 1.0 when provided

Modify: embedding_cluster/indexer.py

• After generating image and text embeddings for each item, apply fusion:

  def fuse_embeddings(
      image_embedding: list[float] | None,
      text_embedding: list[float] | None,
      mode: str,
      weights: list[float] | None = None,
  ) -> list[float]:
      if mode == "concatenate":
          return image_embedding + text_embedding
      elif mode == "average":
          # Requires same dimensionality
          w = weights or [0.5, 0.5]
          return [w[0] * i + w[1] * t for i, t in zip(image_embedding, text_embedding)]
      elif mode == "clip_unified":
          # Both embeddings from CLIP (same space), weighted average
          w = weights or [0.5, 0.5]
          return [w[0] * i + w[1] * t for i, t in zip(image_embedding, text_embedding)]

• When fusion_mode is set, create a single _fused collection instead of separate per-field collections
• For clip_unified mode, use CLIP's text tokenizer + encoder instead of SentenceTransformer
• Store fusion metadata in ChromaDB collection metadata (mode, weights, source fields)

Modify: embedding_cluster/server/models.py

• Add fusion_mode and fusion_weights to IndexRequest
• Add validation for fusion_weights

Modify: embedding_cluster/server/routes/index.py

• Pass fusion parameters to indexer

Frontend

Modify: frontend/src/pages/IndexPage.tsx

• Show fusion options only when both IMAGE and TEXT embedding fields are configured
• Fusion mode selector: None, Concatenate, Average, CLIP Unified
• Weight sliders (visible for Average and CLIP Unified modes): two linked sliders that sum to 1.0
• Info tooltip explaining each fusion mode

Modify: frontend/src/types/index.ts

• Add fusion types to index request interface

Considerations

• Concatenation increases vector dimensionality (896 for CLIP+BGE), which affects ChromaDB storage and distance computation. Document this trade-off.
• CLIP Unified requires loading the CLIP text encoder in addition to (or instead of) SentenceTransformer. The CLIP text encoder produces 512-dim vectors aligned with CLIP image vectors.
• Average only works when both modalities have the same dimensionality. Validate and error clearly if they differ.
• Fused collections should use a distinct naming convention (e.g., prefix_fused_imageUrl_productDisplayName)

Testing Requirements

Full-coverage tests are required for all new code.

Backend Tests

New file: tests/test_fusion.py

• Test fuse_embeddings() with concatenate mode (output length = sum of input lengths)
• Test fuse_embeddings() with average mode (output length = input length, values are weighted means)
• Test fuse_embeddings() with clip_unified mode
• Test custom weights are applied correctly
• Test weights validation (must sum to 1.0)
• Test error when average mode used with different-length vectors
• Test error when one embedding is None in non-optional mode
• Test default weights (0.5, 0.5)

Modify: tests/test_indexer.py

• Test indexing with fusion_mode=concatenate creates fused collection
• Test indexing with fusion_mode=average
• Test indexing with fusion_mode=clip_unified uses CLIP text encoder
• Test collection metadata includes fusion info
• Test fused collection naming convention

Modify: tests/test_settings.py

• Test fusion_mode setting parsing
• Test fusion_weights validation

Frontend Tests

• Index form: fusion controls appear only when both image and text fields set
• Weight sliders: sum to 1.0, linked behavior
• Mode descriptions render correctly

Acceptance Criteria

• User can select fusion mode when both image and text fields are configured
• Concatenation produces embeddings of combined length
• Weighted average produces embeddings with correct weighted values
• CLIP Unified uses CLIP text encoder for text embeddings
• Fused embeddings are stored in a dedicated ChromaDB collection
• Collection metadata records fusion mode and weights
• All new backend code has full test coverage
• Frontend components have tests

Notes

• After implementation, update README.md to document fusion modes (add to Features list, Index parameter table, explain trade-offs)
• This feature is backend-heavy — the frontend changes are primarily in IndexPage
• Consider a future enhancement: learned fusion (training a small projection layer) — out of scope for this issue