TLK-1249 and TLK-1250 - LlamaIndex file tool and documentation improvements

[EugeneLightsOn]

LlamaIndex file tool and documentation improvements

AI Description

This PR introduces changes to the configuration and setup documentation, as well as updates to the src/backend/chat/custom/custom.py and src/community/tools/llama_index.py files.

Configuration and Setup Documentation

• The docs/config_details/config_description.md file is added, providing an explanation of the configuration parameters.
• The docs/setup.md file is updated to include a link to the configuration parameters explanation.

src/backend/chat/custom/custom.py

• The get_managed_tools function is modified to accept an optional full_schema parameter.
• The call_chat function is updated to use the get_managed_tools function with full_schema=True to retrieve managed tools with their full schema.
• The call_chat function now filters tools based on their category, specifically tools with the Category.FileLoader category.
• The update_chat_history_with_tool_results function is modified to remove tools based on their category, rather than their name.

src/community/tools/llama_index.py

• The LlamaIndexUploadPDFRetriever class is updated to use the VectorStoreIndex class for indexing and retrieval.
• The call method is modified to accept a ctx parameter and use it to retrieve files and user information.
• The call method now checks for the presence of a query and files before proceeding with retrieval.
• The call method retrieves files based on their IDs and user ID, ensuring that only relevant files are processed.
• The call method uses the StringIterableReader to load file content, and the SentenceSplitter to split sentences for embedding and indexing.
• The call method now returns an empty list if no files are retrieved or if the query is empty.

New Files

• src/backend/tools/utils/tools_checkers.py: This file contains utility functions for checking tool categories and community tools.
• src/community/config/tools.py: This file defines the CommunityToolName class, which represents community tools and their configurations.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 75.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 80.13%. Comparing base (e6fa042) to head (6f627e3).

Files with missing lines	Patch %	Lines
src/backend/chat/custom/custom.py	80.00%	2 Missing ⚠️
src/backend/tools/utils/tools_checkers.py	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #779      +/-   ##
==========================================
- Coverage   80.13%   80.13%   -0.01%     
==========================================
  Files         238      239       +1     
  Lines       10378    10388      +10     
==========================================
+ Hits         8316     8324       +8     
- Misses       2062     2064       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tianjing-li]

Looks great to me, thank you!