query pipeline

[johnhu33]

query pipeline couldn't retrieve correct node even exact text exists in the index

[dosubot[bot]]

The query pipeline couldn't retrieve the correct node even though the exact text exists in the index because the table parser component was not designed to return relevant rows per table schema. It only returned the table schemas without considering the contextually relevant rows that might contain the exact text.

To address this, the table parser component needs to be expanded to include relevant rows per table schema. This can be done by using vector retrieval of relevant rows based on the original query string.

Here is the expanded table parser component:

from llama_index.core.retrievers import SQLRetriever
from typing import List
from llama_index.core.query_pipeline import FnComponent

sql_retriever = SQLRetriever(sql_database)

def get_table_context_and_rows_str(
    query_str: str, table_schema_objs: List[SQLTableSchema]
):
    """Get table context string."""
    context_strs = []
    for table_schema_obj in table_schema_objs:
        # first append table info + additional context
        table_info = sql_database.get_single_table_info(
            table_schema_obj.table_name
        )
        if table_schema_obj.context_str:
            table_opt_context = " The table description is: "
            table_opt_context += table_schema_obj.context_str
            table_info += table_opt_context

        # also lookup vector index to return relevant table rows
        vector_retriever = vector_index_dict[
            table_schema_obj.table_name
        ].as_retriever(similarity_top_k=2)
        relevant_nodes = vector_retriever.retrieve(query_str)
        if len(relevant_nodes) > 0:
            table_row_context = "\nHere are some relevant example rows (values in the same order as columns above)\n"
            for node in relevant_nodes:
                table_row_context += str(node.get_content()) + "\n"
            table_info += table_row_context

        context_strs.append(table_info)
    return "\n\n".join(context_strs)

table_parser_component = FnComponent(fn=get_table_context_and_rows_str)

This modification ensures that the query pipeline retrieves not only the table schemas but also the relevant rows that might contain the exact text, thereby improving the accuracy of the node retrieval ^[1].

To continue talking to Dosu, mention @dosu.

[johnhu33]

querypipeline couldn't retrieve correct node via retriever even exact text exists in the redis index, how to debug? @dosu

[dosubot[bot]]

To debug the query pipeline when it couldn't retrieve the correct node via the retriever, even though the exact text exists in the Redis index, you can follow these steps:

1. Check the Logs: Ensure that logging is enabled and check the debug logs for any messages that indicate why a node was not selected. The code contains several logging statements that can help you understand the decision-making process at each level.

2. Verbose Mode: Enable verbose mode to get more detailed output. This can be done by setting the verbose parameter to True when initializing the TreeSelectLeafRetriever.

3. Inspect Node Selection: The _select_nodes method is responsible for selecting nodes based on the query. Check the debug logs generated in this method to see which nodes were considered and why certain nodes were not selected.

4. Check Query Templates: Ensure that the query templates (query_template and query_template_multiple) are correctly formatted and appropriate for your use case. These templates are used to generate the prompts for the LLM to select nodes.

5. Extracted Numbers: The extract_numbers_given_response function is used to extract node indices from the LLM's response. Ensure that this function is correctly extracting the numbers and that the numbers correspond to valid node indices.

6. Test with Mock Data: Use mock data to test the retriever's behavior. For example, you can create a mock embedding and mock triplet extraction to simulate the retrieval process. This can help you isolate the issue and understand if the problem lies in the embedding, triplet extraction, or node selection process.

Here is a snippet of the relevant code with logging statements that you should check:

selected_nodes = []
for number_str in numbers:
    number = int(number_str)
    if number > len(cur_node_list):
        logger.debug(
            f">[Level {level}] Invalid response: {response} - "
            f"number {number} out of range"
        )
        continue

    # number is 1-indexed, so subtract 1
    selected_node = cur_node_list[number - 1]

    info_str = (
        f">[Level {level}] Selected node: "
        f"[{number}]/[{','.join([str(int(n)) for n in numbers])}]"
    )
    logger.info(info_str)
    if self._verbose:
        print_text(info_str, end="\n")
    debug_str = " ".join(
        selected_node.get_content(metadata_mode=MetadataMode.LLM).splitlines()
    )
    full_debug_str = (
        f">[Level {level}] Node "
        f"[{number}] Summary text: "
        f"{ selected_node.get_content(metadata_mode=MetadataMode.LLM) }"
    )
    logger.debug(full_debug_str)
    if self._verbose:
        print_text(full_debug_str, end="\n")
    selected_nodes.append(selected_node)

return selected_nodes

Additionally, you can refer to the test cases provided in the llama-index-core/tests/indices/knowledge_graph/test_retrievers.py file to understand how the retriever is tested with mock data. For example, the test_retrieve_similarity function tests the retriever's behavior with a mock embedding and triplet extraction:

@patch.object(
    KnowledgeGraphIndex, "_extract_triplets", side_effect=mock_extract_triplets
)
def test_retrieve_similarity(
    _patch_extract_triplets: Any, documents: List[Document]
) -> None:
    """Test query."""
    graph_store = SimpleGraphStore()
    storage_context = StorageContext.from_defaults(graph_store=graph_store)

    index = KnowledgeGraphIndex.from_documents(
        documents,
        include_embeddings=True,
        storage_context=storage_context,
        embed_model=MockEmbedding(),
    )
    retriever = KGTableRetriever(index, similarity_top_k=2, graph_store=graph_store)

    # returns only two rel texts to use for generating response
    # uses hybrid query by default
    nodes = retriever.retrieve(QueryBundle("foo"))
    assert (
        nodes[1].node.get_content()
        == "The following are knowledge sequence in max depth 2"
        " in the form of directed graph like:\n"
        "`subject -[predicate]->, object, <-[predicate_next_hop]-,"
        " object_next_hop ...`"
        "\n['foo', 'is', 'bar']"
    )

By following these steps and inspecting the logs, you should be able to identify why the correct node was not retrieved and make the necessary adjustments ^[1][2].

To continue talking to Dosu, mention @dosu.

[johnhu33]

how to retrieve exact text exists in the redis index, how to debug? @dosu

[dosubot[bot]]

To retrieve the exact text that exists in the Redis index and debug the process, you can use the query method of the RedisVectorStore class. Here is an example of how you can do this:

from llama_index.vector_stores.redis import RedisVectorStore
from llama_index.vector_stores.types import VectorStoreQuery
from redis import Redis

# Initialize the RedisVectorStore
vector_store = RedisVectorStore(
    redis_client=Redis.from_url("redis://localhost:6379"),
    overwrite=True
)

# Create a query object
query = VectorStoreQuery(
    query_embedding=[0.1, 0.2, 0.3, ...],  # Replace with your query embedding
    similarity_top_k=10  # Number of top similar results to retrieve
)

# Query the index
result = vector_store.query(query)

# Retrieve the exact text from the results
for node in result.nodes:
    print(node.text)

In this example, replace [0.1, 0.2, 0.3, ...] with your actual query embedding. The query method will return a VectorStoreQueryResult object, which contains the nodes that match the query. You can then iterate over the nodes and print the text attribute to retrieve the exact text stored in the Redis index.

Additionally, if you want to inspect the raw text stored in the index, you can use the save_to_disk method to save the index to a JSON file and inspect it manually. This can help you understand how the data is stored and debug any issues:

# Save the index to a JSON file
vector_store.save_to_disk("index_dump.json")

# Open and inspect the JSON file
with open("index_dump.json", "r") as file:
    index_data = file.read()
    print(index_data)

This will allow you to see the raw text and other details stored in the index, providing a clearer picture of the data and aiding in debugging.

Troubleshooting

If you get an empty query result, there are a couple of issues to check:

Schema

Ensure that your schema includes the required fields id, doc_id, text, and vector.

Prefix Issues

Make sure that the chosen prefix, as part of the index schema, is consistent across your code.

Data vs Index

If you have an existing index and want to make sure it's dropped, you can run FT.DROPINDEX <name of your index> in the Redis CLI. Note that this will not drop your actual data unless you pass DD.

Empty Queries When Using Metadata

If you add metadata to the index after it has already been created and then try to query over that metadata, your queries will come back empty. Redis indexes fields upon index creation only ^[1][2][3].

To continue talking to Dosu, mention @dosu.