llama_index with LLM doing out of contex...

I think this is more related to LLM side rather than llama-Index side. 🧐

not sure about this. We didn't change anything to our calls to the LLM and are starting to get unwanted out of context responses

Maybe if there are any changes to how to query is structured for LLM then that could be the case.

@Maximus openai seem to update their models frequently without telling anyone. Performance seems to vary quite a bit, which is likely the cause of this

Some people have had luck appending extra instructions to the query

Others have modified the prompt templates

There is solutions 👀 even though the default prompt tells it not to use prior knowledge already 🫠

Thanks @Logan M, but shouldn't this be tamed by the default qa prompt?

Probably! Just haven't had time to mess around with it. Hard to make a prompt that generalizes well to everything and every LLM 😅

I welcome a PR though for this if you are interested!

For the past 1-2 months this has been happening regularly. First chatgpt started becoming weird now the main model itself is wilding 😅

@WhiteFang_Jr I agree, it's pretty annoying tbh. It sucks that the only solution (either in a PR or customizing prompts yourself) is prompt engineering :PSadge:

so tedious

@Logan M If none of the template has been selected by the user, do we have any default template that is used while querying to the LLM ?

Yea, the default templates are here:
https://github.com/jerryjliu/llama_index/blob/main/llama_index/prompts/default_prompts.py

And templates specific to chat models (i.e. gpt-3.5, gpt-4) are here:
https://github.com/jerryjliu/llama_index/blob/main/llama_index/prompts/chat_prompts.py

By and large the refine and text-qa templates are the main ones used

I'll try to give it a go, would be a good way to spend the weekend lol 😅😂

@WhiteFang_Jr that would be awesome! 🙏 🙏

Hey! @Maximus

Are you are using GPT-3 in your services?

Yes

I have been testing the QA prompt on GPT-3 and GPT-3.5 both

Getting more cases of out of context response in GPT-3 case

Isn't that strange? Shouldn't the prompt act the same way?

Yeah it is. Maybe a way to move more users to opt for GPT-3.5. lol jk 😂

By making GPT-3 to act a bit dumb sometimes

Alright. Will move to 3.5 thanks for the follow up. Much appreciated