How do I ask the same query to all documents in the doc...

@kapa.ai How do I ask the same query to all documents in the docstore ?

8 comments

Please stand by @Maximus as I search the knowledge base. I normally take about 20 seconds or so. :wave:

----------------------
To ask the same query to all documents in the docstore, you can use the query engine of LlamaIndex. Here's a step-by-step guide:

First, you need to import the necessary modules and set up your environment. This includes setting up logging and applying nest_asyncio.

Plain Text

import nest_asyncio
import logging
import sys
import os
from llama_index import (
    SimpleDirectoryReader,
    ServiceContext,
    LLMPredictor,
    StorageContext,
)
from llama_index import VectorStoreIndex, SummaryIndex, SimpleKeywordTableIndex
from llama_index.composability import ComposableGraph
from llama_index.llms import OpenAI
from llama_index.response.notebook_utils import display_response

nest_asyncio.apply()
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

Load your documents and parse them into nodes.

Plain Text

reader = SimpleDirectoryReader("../paul_graham_essay/data")
documents = reader.load_data()

from llama_index.node_parser import SimpleNodeParser
nodes = SimpleNodeParser.from_defaults().get_nodes_from_documents(documents)

Add your documents to the docstore. In this example, we're using a MongoDB docstore.

Plain Text

MONGO_URI = os.environ["MONGO_URI"]

from llama_index.storage.docstore import MongoDocumentStore
from llama_index.storage.index_store.mongo_index_store import MongoIndexStore

storage_context = StorageContext.from_defaults(
    docstore=MongoDocumentStore.from_uri(uri=MONGO_URI),
    index_store=MongoIndexStore.from_uri(uri=MONGO_URI),
)

storage_context.docstore.add_documents(nodes)

Define your indexes. Each index uses the same underlying Node.

kkapa.ai

Plain Text

summary_index = SummaryIndex(nodes, storage_context=storage_context)
vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
keyword_table_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)

Now you can use the query engine to ask the same query to all documents in the docstore. Here's an example using the summary_index:

Plain Text

query_engine = summary_index.as_query_engine()
response = query_engine.query("Your query here")
display_response(response)

You can replace "Your query here" with your actual query. The same process can be repeated for vector_index and keyword_table_index as well.

Sources:

MongoDocstoreDemo.ipynb ()
RedisDocstoreIndexStoreDemo.ipynb ()

----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------

LLogan M

Do you want a single response, or a response for each?

MMaximus

a separate response for each

LLogan M

Use index.as_query_engine(response_mode="accumulate") for listindex/summaryindex

MMaximus

works great ! Thanks

MMaximus

looping over nodes was not working so well!

Add a reply

Find answers from the community

How do I ask the same query to all documents in the docstore ?