How do I ask the same query to all documents in the doc...

At a glance

The community member asked how to query all documents in a docstore. The response provided a step-by-step guide using the LlamaIndex library, including setting up the environment, loading documents, adding them to a MongoDB docstore, defining indexes, and using the query engine to execute the same query across all documents. The community members discussed using different index types and response modes to get the desired results.

MMaximus

@kapa.ai How do I ask the same query to all documents in the docstore ?

8 comments

kkapa.ai

Please stand by @Maximus as I search the knowledge base. I normally take about 20 seconds or so. :wave:

kkapa.ai

----------------------
To ask the same query to all documents in the docstore, you can use the query engine of LlamaIndex. Here's a step-by-step guide:

First, you need to import the necessary modules and set up your environment. This includes setting up logging and applying nest_asyncio.

Plain Text

import nest_asyncio
import logging
import sys
import os
from llama_index import (
    SimpleDirectoryReader,
    ServiceContext,
    LLMPredictor,
    StorageContext,
)
from llama_index import VectorStoreIndex, SummaryIndex, SimpleKeywordTableIndex
from llama_index.composability import ComposableGraph
from llama_index.llms import OpenAI
from llama_index.response.notebook_utils import display_response

nest_asyncio.apply()
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

Load your documents and parse them into nodes.

Plain Text

reader = SimpleDirectoryReader("../paul_graham_essay/data")
documents = reader.load_data()

from llama_index.node_parser import SimpleNodeParser
nodes = SimpleNodeParser.from_defaults().get_nodes_from_documents(documents)

Add your documents to the docstore. In this example, we're using a MongoDB docstore.

Plain Text

MONGO_URI = os.environ["MONGO_URI"]

from llama_index.storage.docstore import MongoDocumentStore
from llama_index.storage.index_store.mongo_index_store import MongoIndexStore

storage_context = StorageContext.from_defaults(
    docstore=MongoDocumentStore.from_uri(uri=MONGO_URI),
    index_store=MongoIndexStore.from_uri(uri=MONGO_URI),
)

storage_context.docstore.add_documents(nodes)

Define your indexes. Each index uses the same underlying Node.

kkapa.ai

Plain Text

summary_index = SummaryIndex(nodes, storage_context=storage_context)
vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
keyword_table_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)

Now you can use the query engine to ask the same query to all documents in the docstore. Here's an example using the summary_index:

Plain Text

query_engine = summary_index.as_query_engine()
response = query_engine.query("Your query here")
display_response(response)

You can replace "Your query here" with your actual query. The same process can be repeated for vector_index and keyword_table_index as well.

Sources:

MongoDocstoreDemo.ipynb ()
RedisDocstoreIndexStoreDemo.ipynb ()

----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------

LLogan M

Do you want a single response, or a response for each?

MMaximus

a separate response for each

LLogan M

Use index.as_query_engine(response_mode="accumulate") for listindex/summaryindex

MMaximus

works great ! Thanks

MMaximus

looping over nodes was not working so well!

Add a reply

Find answers from the community

How do I ask the same query to all documents in the docstore ?