----------------------
To load only the necessary data based on a query in LlamaIndex, you would typically use a query engine. The query engine is responsible for selecting the relevant data based on the query.
For example, in the Joint QA Summary Query Engine, the router query engine selects the relevant query engine based on the context of the documents needed to answer the question. This is demonstrated in the following code snippets:
INFO:llama_index.query_engine.router_query_engine:Selecting query engine 0 because: This choice is relevant because it involves retrieving specific context from documents, which is needed to answer the question about the author's activities growing up..
INFO:llama_index.query_engine.router_query_engine:Selecting query engine 0 because: This choice is relevant because it involves retrieving specific context from documents, which is needed to answer the question about the author's activities in art school..
In these examples, the query engine is selecting the necessary data (documents) based on the query about the author's activities.
For loading data, you can use the
SimpleDirectoryReader to load data from a local directory as shown in the following code:
from llama_index import SimpleDirectoryReader
# try loading great gatsby
documents = SimpleDirectoryReader(
input_files=["./gatsby_full.txt"]
).load_data()In this example, the
SimpleDirectoryReader is used to load the data from the file
gatsby_full.txt.
