- You'll have to modify the text qa and refine prompts with more specific instructions. ChatGPT is very stubborn sometimes lol
- Llama index does not keep track of any chat history. You'll need langchain + Llama Index as a tool for that
- System prompts can be added by customizing the prompt templates (see the FAQ in the pinned messages, you'll see what I mean) or you can prepend a system message to every prompt using the code from the attached screenshot
@Logan M 2 and 3 make sense. But for one, is there no option for terminating the current session and starting a new one?
Technically, there is no "current session", every api call to openai is a clean slate
??? then how does it know what it says?
it keep saying i already gave this answer. Refer to that. And i dont see anything else in the response
im so confused. Maybe its part of the prompt and it get it from that?
Check out the text qa and refine templates in the repo. It will make more sense.
Once we have an initial answer, we send new context + existing answer + query and ask it to update or repeat its original answer
i see. So I can better understand why it says what it says by looking at the values for those templates?
Does the chat prompt also get replaced by query config kwargs? Also side note empirically it seems like chatgpt pays more attention to human messages than system messages, so if you're not getting good results, could also add instructions and guardrails prepended as human messages instead
Yea if you set the prompt templates in the query config kwargs, they will get used for the indexes that config applies to
is it still replacing "refine_template"? just switch that with a new chatgpt specific prompt template?
Yup! Let me know if you have troubles. It should be working
Take a look at this
"The original answer is still accurate and complete given the new context. No additional information or refinement is needed. The provided context refers to the overall structure of the document and reporting requirements, but does not include any new information on deadlines."
@Logan M like what is the original answer? After I press run in jupyter notebook i dont see the original answer just this.
Right. This is a "bug" with chatGPT sometimes.
It gives an initial answer using the text_qa_template
But, usually we need to look at more text (maybe your top k is 3, maybe it's a list index, etc).
So it moves to the next piece of text and says "hey, I asked this and have an existing answer. Can you make the answer better using this new context? If not, return the original answer" (highly paraphrasing here lol)
But, chatgpt sometimes struggles to understand this instruction fully. I know, it's painful.
Hopefully some upcoming changes will help with this. Otherwise, it's a prompt engineering problem for now :PepeHands:
oof this is not ideal. I'll see how far i can get by just prompt eng
Like, it usually works. Or at least it used to. OpenAI has no version control on their models, so the model for gpt-3.5-turbo could also have been tweaked
I dont know the performance is severly shot in gpt-3.5 Take a look at this very simple prompt
response = index_list[1].query(
"What is the program title for this proposal?",
mode="embedding",
similarity_top_k=1,
#service_context=service_context,
response_mode="tree_summarize",
)
print(response)
The program title for this proposal is "XYZ University Program Solicitation".
Now if i activate service context and use gpt-3.5 i get: The program title is not provided in the given context information.
Someone had this exact issue and i was able to solve it. But this one seems extremely absurd...
Top k = 1 can still result in multiple refines. If the chunk size + prompt + query is bigger than 4097, it will split the context into more chunks and refine
Try setting chunk_size_limit=2000 in the service context too before constructing the index?
I tried changing chunk size limit and almost everything i possibly could
I havent rtied this in langchain but i think something is wrong with llama index here.
3.5 will not you give you an answer to matter how you try.
you agree that something isnt working right?
Oh totally. I just don't think it's llama index's fault 😅
So just to be clear this example would have worked correctly before without the context provided garbage?
I mean, yes. But my guess is something has changed recently with the gpt-3.5 model, and now the prompts we have inside llama index for chatgpt are 💩
Like this example, and a few others I have, worked fine a few days ago. Nothing changed in llama index that would cause this 🙃
ok cool lets wait and see what open ai says/does in the next few days.
If not i might have to pull up to their office and ask whats going on haha
