Find answers from the community

Updated 2 years ago

When querying OpenAI gpt 3 5 turbo I m

When querying OpenAI gpt-3.5-turbo I'm getting back responses that include "The provided context is not relevant to the original question" and "Therefore, the original answer remains the same:" followed by an answer. Can someone direct me on how to stop that? I'm using latest Lllamindex with
Plain Text
 llm_predictor = LLMPredictor(llm=ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo", max_tokens=2048))
    service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, chunk_size=1024)
a
e
5 comments
What is your goal? The reason its coming up with this response is likely that the data you have ingested does not have information sufficient to answer the question, and this is a really nice default as it shows that you can restrict the LLM and not have hallucination problems. One question is, did you expect or are you sure that your ingested data should have an answer? If you think your data does have the answer, then you would want to investigate why the answer is not found in your data. If you want the LLM to augment your private data with its own knowledge, then you would want to modify the prompt per the customizing prompts section in the docs. If you want to augment your private data with something like web search, you could look at the langchain integration and chain on a web search step to improve upon the initial response, which would probably only make sense if the majority of questions for your use case actually did get valuable context from your private data.
one simple thing you could try is upping your temperature, but it all depends on the details of what you are trying to accomplish
@afewell My apologies. I should have clarified that I am not seeing the previous responses. I believe the LLM is being queried multiple times to refine the response so what I wrote above is all I'm getting form an initial question. I'm not using any refine templates so I didn't think this was the case? For additional context this is my query engine syntax:
Plain Text
query_engine = index.as_query_engine(similarity_top_k=3, text_qa_template=QA_PROMPT)
    response = query_engine.query(query_str)
and the qa prompt is the taken straight from the docs. I appreciate the assist!
ah yes by default it will query and refine with all matching nodes, or in your case the top 3, and only show you the final result of that. I think you can see each progression with logging level but I havent tried myself, one other method could be to use the accumulate response mode "accumulate: Given a set of Node objects and the query, apply the query to each Node text chunk while accumulating the responses into an array. Returns a concatenated string of all responses. Good for when you need to run the same query separately against each text chunk."
I will try that. Thank you @afewell
Add a reply
Sign up and join the conversation on Discord