For some reason, making LLM queries with llama-index VectorStoreIndex seems to be giving worse results than just plain LLM.
For instance, when I do resp = llm.complete("What do you know about the Chord P2P protocol? Talk specifically about the algorithms known as P-Grid and M-Chord") I get a detailed answer that is basically correct, well-written, and describes the algorithms in question in appropriate detail.
However, when I ask exactly the same question, using exactly the same LLM (mistral 7B) using streaming_response = query_engine.query("What do you know about the Chord P2P ...) I get a useless answer that mostly just told me the title and authors of the Chord paper that I ingested into my VectorStoreIndex, and then says that "details of M-Chord and P-grid cannot be provided without further context or prior knowledge"
I don't understand (a) why it "loses" the prior knowledge that the LLM clearly already had about these two algorithms when I try to ask the question with query_engine and (b) why it is not pulling any information from the PDF in the index that extensively talks about both M-Chord and P-Grid
When you just call a LLM you wont have the same prompt templates. With the query engine there is a template that gets sent that outlines the response should be based on the ingested documents. You can modify the prompt template to give some leniency towards using outside knowledge in addition to documents
OK, thanks I'll mess around with some different prompt templates. Is there a way to see which prompt template is currently being used by the query_engine?
