LlamaIndex

Llama Index works fine, but when using custom models you need to take a lot of attention to ensure the prompts are formatted correctly

See this thread

Llama2 has very specific requirements for prompting

Yes I see ok

For the moment,

I used to load the llm not the custom llm module from llama index

but the one advised in this repository:

so it's a GPTQ instance loaded with auto_gptq

and the huggingface pipeline

so basically no need of this

I think

the problem I have is that the prompt sended to the llm is talking about sources, and the llm answer also with "sources", even if there is only 1 text (it's sources of chunk I think)

Are you using tree_summarize? That's the only mode that mentions sources in the prompt (because it's summarizing chunks from multiple sources)

hi @Logan M

No I'm only using this function from the llamaindex doc:

But there is indeed Citation in the name of the module:

Finally, I don't understand why the message is truncated

ah, the citation query engine also mentions sources in the inputs. It treats each text chunk as a source, and attempts to prompt the LLM to write in-text citations

It's truncated because by default llama-index only leaves room for 256 tokens. Additionally, lots of models also default to stopping at 256 output tokens

So you can change the model config, as well as set num_outputs=300 or similar in the service context 🤔