what happens if you run print(len(docstore.docs))
?
oh wait, I think I know the real issue
You need to actually "load" the index from the index store, since it can technically contain many indexes
index = load_index_from_storage(storage_context, service_context=service_context)
If you have many indexes in the same index store, you need to set and use a specific index_id
as well
hmm, ok, I'll try that. I separate indexes into mongo collections, so its all 1 to 1...
I want to use store too. I think it will speedup load time
That was it. Thank you @Logan M Logan!! Now, how do we use a SummaryIndex to retrieve an entire document? From what I've read about SummaryIndex, its supposed to be the index to use when you want to summarize an entire document. So I have a bunch of long documents (more than 4 nodes lets say) and I want to output the bulk of the ref_doc, what is the approach for that?
I really like using Mongo its very intuitive and easy to query. highly recommend
Its better than filesystem?
in filesystem I can put books I want
much better! the key is to figure out the metadata before you start creating documents and nodes. the more metadata you have the easier it is to query/filter the documents
@theta do you use jupiter notebook to speedup testing? python imports at start are so slow for me. even on ssd
yes, all development is in jupyter lab notebooks
Once the document is inserted into an index, the overall document is actually lost.
It is chunked into nodes. I would set metadata on documents so that the metadata shows up on the nodes, so that you can find the overall document later?
I totally understand what you're saying. I think I worded it poorly. what I'm trying to say is, every week I get a new list of documents, and I'm trying to develop a pipeline that allows the LLM to evaluate each document and classify it and then write a short summary if its a long document and its the "how to get LLM to evaluate the entire document" portion I'm trying to understand, so I thought I needed to build the SummaryIndices so that the LLM could traverse the entire document. with VectorIndex, we're just doing semantic similarity on a query, but I want to have the LLM evaluate the entire document and come up with a summary.
Oh ok! Then yea, you can just query the summary index, and the LLM will read the every node in the index
Usually, I like to set the response mode to tree_summarize and also enable async for speedups
query_engine = index.as_query_engine(response_mode='tree_summarize', use_async=True)
response = query_engine.query("Summarize and evaluate this text for XXX")
If I try to load LLM using Jupyter I get an error UnsupportedOperation: fileno
Have you faced this?
from llama_index.llms import OpenAI
import openai
openai.api_key="sk..."
llm = OpenAI(temperature=0.1, model="gpt-3.5-turbo", max_tokens=1000)
this works for me
@Logan M what can we do if the SummaryIndex.query_engine() doesn't seem like its parsing the entire document? In one of the documents I added to the SummaryIndex is a heading and value combination I added to the query " identify if it has an 'Official Fix' under the heading 'Temporal score metrics'". Now I'm looking at the document source and the document I passed to .from_documents() and the heading+Official Fix strings are in there, but they are way down. Is this a matter of poor prompt or not enough tokens (I limit to 1000 tokens in service_context) or I need to do something else for long documents to ensure that the LLM scans to the end?
The LLM will read everything in the index no matter the context window, but it might be across multiple LLM calls under the hood