[Question]: BETTER PRATICES FOR PRODUCTS ANALYSIS AND RECOMMENDATION

[hungphongtran-pixta]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Purposes:

I have a database of smartphone products, which have been crawled from the Amazon. Each product contains brand, feature bullets, reviews and price. I am building a LlamaIndex integration which takes an arbitrary user input (hobbies, preferences, ...) and output the product recommendation.

Current Pratices:

Setup:

• The database is in CSV format.
• SimpleCSVReader is used to parse the CSV into Documents and to Indexes.
• I use default Chat Engine powered by GPT 3.5 API, taking user data as inputs and output product recommendation.

Output:

• Chat Engine can reason over user data to output products as well as the reason why to suggest these products based on their features and reviews.
• Chat Engine cleaned and shortened the name of the products (being chaotic in the database).
• However, as the name being cleaned, Query Engine/Manual Search is not available to restore the product price.

Questions:

• What are the better pratices or tips to store data instead of CSV, config the Chat Engine and Query Engine for later metadata search?
• What might be the method to do query boosting in which I would be able to boost a certain products in the Chat Engine output?
• What is some recommended approach to update new data into the database without re-embedding the whole dataset?

Thank you guys a lot! Many thanks to the creators and the community of LLamaIndex! You guys are so lit!

[dosubot[bot]]

🤖

Hello,

Thank you for your detailed question and for using LlamaIndex! I'll try to address your questions one by one.

1. Data Storage: As you're currently using CSV for data storage, you might want to consider using a database system for more efficient data handling and querying. LlamaIndex supports both SQL and NoSQL databases. For SQL databases, you can use the SQLDatabase class, which provides helper utilities for insertion and querying. For NoSQL databases, LlamaIndex provides several classes for handling data stored in different types of NoSQL databases. These include FirestoreDocumentStore for Google Firestore, SimpleDocumentStore for simple document databases, MongoDocumentStore for MongoDB, KVDocumentStore for key-value stores, and RedisDocumentStore for Redis. Each of these classes provides methods for interacting with the respective NoSQL database. You can find more details in the source code here and here.

2. Chat Engine and Query Engine Configuration: For configuring the Chat Engine and Query Engine, you can use the ChatEngine and QueryEngine classes respectively. The ChatEngine class provides methods for setting up the chat engine, including setting the model, tokenizer, and device. The QueryEngine class provides methods for setting up the query engine, including setting the retriever and reader. You can find more details in the source code here and here.

3. Query Boosting: For query boosting, you might want to consider using the BoostingQuery class in Elasticsearch. This class allows you to boost the relevance score of documents matching a positive query while reducing the score of documents matching a negative query. You can find more details in the Elasticsearch documentation here.

I hope this helps! If you have any further questions, feel free to ask.

Best,
Dosu

Sources

• llama_index/readers/file/base.py
• llama_index/langchain_helpers/sql_wrapper.py
• llama_index/storage/docstore/init.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.