Hi guys, Sometimes when I generate using

llama2 has decided to freak out it seems lol

Something about the text it was reading maybe caused this

I've already done several things, I used the sentence splitter and left my paragraphs separated by \n\n\n, and I've left them with just 1 \n. However, there is always some example that gives this altered result

Could downgrading the version help?

I don't think this is related to any version. More like something to do with the input text, or maybe improper prompt formatting, is causing issues

Which llm class is this ? How did you set it up?

I'll create a colab notebook, can you do help me to understand this?

I`m using CustomLLM for connect in my api that host Llama2, and BaseEmbedding to connect another api that host BGE-base

My system prompt is:

system_prompt = """<|SYSTEM|>Context information is below.
---------------------
{context_str}
---------------------
Given context information and not prior knowledge, respond to the user last message contained in the ChatSession.
ChatSession: {query_str}
Answer:
"""

and my query is:

response = query_engine.query("""
Assistant: how are you doing today?
User: Hi, I'm doing well. I'm interested in purchasing a new laptop.
Assistant: That's great to hear! I'd be happy to assist you with that. Have you considered the our account as a mode of payment?
User: No, I haven't.
""")

I`ve the .txts inside the data folder and the struct is:

User: No, pay pal account Assistant: I recommend the our account because...

User: .... Assistant: I recommend the our account because...

User: .... Assistant: I recommend the our account because...

....

....

this is not look like a llama prompt template to me. Was model are you using?

I'm using the llama-2-13b-chat for the generation and bge-base-en-1.5.

https://colab.research.google.com/drive/1dyR_C5pHsE-X72b-k-vA6LUJAf8ggTm9#scrollTo=u73iZkWPaQo4 This notebook contains the code used, it is not running because I use the models via API and cannot access them via Colab

just looking at the model card of llama-2-chat you are not using the right prompt template.
https://huggingface.co/TheBloke/Llama-2-13B-chat-GPTQ

small models like that dont like that at all and start spitting random stuff 😉

OH MY GOD