[RFC] Add initial support for Genie

[bhavnicksm]

Hey there, folks! 😄

We want to add support for a class called Genie that would allow for interfacing with generative LLMs or API calls to them inside Chonkie. This functionality is essential to add ContextualRefinery (#105) but not only that, it would be important for AgenticChunking, LumberChunking and other generative chunking approaches. These chunking approaches are usually more time-consuming but give higher-quality text chunks than rule-based chunking approaches, making them a nice addition to have!

To support Genie, we would need a BaseGenie class that would define what every Genie from there on should implement.

Additionally, since Genies are meant to be easy to "plug in" into the ContextualRefinery or other chunkers mentioned above, they should come with a supporting class Library (which in turn would have it's ownBaseLibrary to define what to implement).

Each Library would hold a set of template prompts in the f-string format, along with some way to define what the f-string requires. (OFD)

An additional utility method inside Library that can create a prompt based on the specific **kwargs passed into them and the prompt name would be a nice addition to quickly see the prompt from the template.

Generally, we would want the Library to come with set defaults, but in case the user wishes to change the prompt and the template for any action, we should allow them to do so by initializing a Library, picking the category they want to set as, and passing it into the Genie for it to interface with.

Since Chonkie defines its tasks, whenever a new task is called that needs a Genie, we would add the necessary default prompts inside the Library based on our own tests so the user either get's a verified prompt or nothing at all. In case a prompt template is not present in the Library it can throw an error saying something along the lines of This genie has no prompt defined for {task}; Add prompt or use a supported genie.

Generally, these are my current thoughts on the implementation of the Genie. Of course, open to discussions on it!

Thanks! 😊

[bhavnicksm]

Hey @Udayk02 👋

Thanks for the interest in this feature; we can discuss how to go about this feature support here!

Let me know what you think about it and, generally, if you have any comments on implementation~

Thanks 😊

[Udayk02]

To my understanding,

Genie is something sorta a wrapper that chonkie can use to communicate with different LLMs. For example, OllamaGenie will communicate with ollama. OpenAIGenie will communicate with openai.
Now, Library is something like a library of templates. A certain template is something we will use in chunking based on its use-case. For example, templates would require a set of inputs (chunks), prompts, etc. to facilitate overlaps presumably in ContextualRefinery. Template guides the llm to provide us with the proper context. I believe, whatever the output, that has to be heavily post-processed in order to maintain the proper chunk_size, or other metrics that we have to follow in the active chunking process.

Going forward, we can introduce custom templates, custom prompts (idk), etc to make it more comfortable for the user.

Please provide suggestions and a pipeline would be appreciated so that I will pitch in where I can 😄
Excited to work on this!

Thanks!

[bhavnicksm]

Hey @Udayk02,

Genie is something sorta a wrapper that chonkie can use to communicate with different LLMs. For example, OllamaGenie will communicate with ollama. OpenAIGenie will communicate with openai.

Yes, that's accurate! That's exactly what we want Genie for.

Now, Library is something like a library of templates. A certain template is something we will use in chunking based on its use-case. For example, templates would require a set of inputs (chunks), prompts, etc. to facilitate overlaps presumably in ContextualRefinery. Template guides the llm to provide us with the proper context. I believe, whatever the output, that has to be heavily post-processed in order to maintain the proper chunk_size, or other metrics that we have to follow in the active chunking process.

For the ContextualRefinery the prompt would only need the full_text and the chunk to know the exact context to add to the chunk to enrich it!

Please provide suggestions and a pipeline would be appreciated so that I will pitch in where I can 😄
Excited to work on this!

Yes, every Genie should have a .generate function that would take in a prompt and generate the response. Other than that, we can start with a BaseLibrary and a BaseGenie class that I can add on. Then, we can add support for ClaudeGenie and GPTGenie/OpenAIGenie.

Currently, I'm preoccupied with the next v0.5.0 release, but I can make the necessary base classes to work with soon. From there on, it would become simpler to support more Genies and add a ContextualRefinery.

Thanks! 😊

[Udayk02]

Sure! No problem. Looking forward to it!

[Udayk02]

Just to mention here: adding to the genie part, we have to leverage the features such as "prompt caching" or "context caching" if we are proceeding with the implementation of features like ContextualRefinery.