Infinity embeddings

[canesin]

Add Support for Infinity Embeddings, JSON response and pre-defined IDs on the API.

Overview

This PR adds support for local embedding generation using the Infinity Embedding library with a focus on the high-quality Snowflake Arctic Embed models.

Changes

1. New Embedding Backend

• Added a new embedding backend infinity that uses the infinity-embed package
• Allows local embedding generation using Hugging Face models without API calls
• Default model is the top-performing "Snowflake/snowflake-arctic-embed-l-v2.0"

2. Model Caching

• Implemented caching to load models only once and reuse them for subsequent embeddings
• Prevents inefficient reloading of models on every embedding request

3. Asymmetric Embedding Support

• Separate query embedding functions in server configuration when using Snowflake models
• This follows Snowflake's specifications for optimal retrieval results

4. Error Handling

• Implemented retry logic for embedding generation with specific exception types
• Handles connection, timeout, and value errors appropriately

5. Example Implementation

• Added a demo script lightrag_infinity_demo.py showing how to use Infinity embeddings
• Demonstrates both document and query embedding with the asymmetric retrieval pattern

6. Configuration

• Updated environment example and argument parsing to support the new embedding binding
• No additional host configuration needed as embeddings are generated locally

Dependencies

• Added infinity-emb[all] to requirements.txt

Testing

The implementation has been tested using the Snowflake Arctic Embed model for both standalone usage and integration with the LightRAG API server.

[canesin]

I am opening the PR here to contribute back and also stop a bit the divergence of our fork main branch. This includes also the given ID changes and JSON response changes.
Will add some testing and example for both before removing the draft status.