Expose a LLamaIndex application to GUIs like LibreChat using OpenAI API spec

[PierreMesure]

Hi,

I've been using LlamaIndex to build a few applications and I'm currently using FastAPI in order to expose an API, as well as a simple Vite frontend that uses this API.

I wonder if there are existing solutions to expose an API that matches OpenAI API specifications. I'm trying to migrate towards products such as LibreChat, OpenWebui and they are compatible with many LLM providers using this de facto standard:

generally every OpenAI API-compatible service should work

Source: LibreChat docs

Do you know of any "proxy" that would act as an intermediate layer between LlamaIndex and this API standard?

[dosubot[bot]]

Hello @PierreMesure! I'm here to help you with any bugs, questions, and contributions you have regarding the repository. I'm an AI bot designed to assist you while you wait for a human maintainer.

I found relevant information that might help you. The LlamaAPI class in the LlamaIndex repository acts as an intermediate layer (or "proxy") between LlamaIndex and the OpenAI API standard. It uses the llamaapi client to interact with the OpenAI API, converting messages and responses between the formats used by LlamaIndex and OpenAI.

Here is the relevant part of the code:

class LlamaAPI(CustomLLM):
    # ... other fields and methods ...

    def __init__(
        self,
        model: str = "llama-13b-chat",
        temperature: float = 0.1,
        max_tokens: int = DEFAULT_NUM_OUTPUTS,
        additional_kwargs: Optional[Dict[str, Any]] = None,
        api_key: Optional[str] = None,
        callback_manager: Optional[CallbackManager] = None,
        system_prompt: Optional[str] = None,
        messages_to_prompt: Optional[Callable[[Sequence[ChatMessage]], str]] = None,
        completion_to_prompt: Optional[Callable[[str], str]] = None,
        pydantic_program_mode: PydanticProgramMode = PydanticProgramMode.DEFAULT,
        output_parser: Optional[BaseOutputParser] = None,
    ) -> None:
        try:
            from llamaapi import LlamaAPI as Client
        except ImportError as e:
            raise ImportError(
                "llama_api not installed."
                "Please install it with `pip install llamaapi`."
            ) from e

        self._client = Client(api_key)

        super().__init__(
            model=model,
            temperature=temperature,
            max_tokens=max_tokens,
            additional_kwargs=additional_kwargs or {},
            callback_manager=callback_manager,
            system_prompt=system_prompt,
            messages_to_prompt=messages_to_prompt,
            completion_to_prompt=completion_to_prompt,
            pydantic_program_mode=pydantic_program_mode,
            output_parser=output_parser,
        )

    @llm_chat_callback()
    def chat(self, messages: Sequence[ChatMessage], **kwargs: Any) -> ChatResponse:
        message_dicts = to_openai_message_dicts(messages)
        json_dict = {
            "messages": message_dicts,
            **self._model_kwargs,
            **kwargs,
        }
        response = self._client.run(json_dict).json()
        message_dict = response["choices"][0]["message"]
        message = from_openai_message_dict(message_dict)

        return ChatResponse(message=message, raw=response)

This class handles the conversion of messages to and from the OpenAI format and manages the interaction with the OpenAI API through the llamaapi client ^[1].

To continue talking to Dosu, mention @dosu.

[PierreMesure]

I've been looking some more since I posted this and I really can't find a ready-made solution!

Here's the closest I found:

• How to build an OpenAI-compatible API
• How to Build an OpenAI-Compatible API using Any FastAPI Application with LLM Integration

I think another elegant solution could also be to build on a tool that focuses on exposing an OpenAI-compatible API. LiteLLM and its CustomLLM class seems to be the best candidate.

EDIT: I found LitServe which offers the same possibilities as LiteLLM and gives code examples with LlamaIndex! Exposing an OpenAI-compatible API seems to be as easy as specifying it with a parameter! I haven't tried yet but this looks super promising! Insane that it took so much googling to learn about Litserve! 🤯🤯🤯

EDIT: I made it work, here is the code.

[joreilly86]

Very nice @PierreMesure - I have been struggling with this.