Find answers from the community

Updated last year

The OpenOrca Platypus2 13B model

The OpenOrca-Platypus2-13B model supports this template for chat:
Plain Text
<|user|> <|user-message|><|end_of_turn|>\n<|bot|> <|bot-message|>\n


But llamaindex react loop agent dont use this format. there is many \n and examples like '''Please use valid JSON.......
Observation: tool response....
...

Can it be really difficult for custom LLM?How to fix?
L
А
11 comments
How did you setup your LLM?
index.as_query_engine gives better result, but it cant tell me the whole story of the book (very big docx file). It tell only part. And react loop make only 1 step.

index.as_chat_engine gives completely wrong answer. and very short.
I think this is because in chat mode it use messages_to_prompt routine. And react loop template not compatible with LLM chat template
Assuming the messages_to_prompt method is correct, it should be working fine 🤔 It will get used by an agent
I have some mistakes in code because prepared to share with you.
message_to_prompt contains:
Plain Text
'\nYou are designed to help with a variety of tasks, from answering questions     to providing summaries to other types of analyses.\n\n## Tools\nYou have access to a wide variety of tools. You are responsible for using\nthe tools in any sequence you deem appropriate to complete the task at hand.\nThis may require breaking the task into subtasks and using different tools\nto complete each subtask.\n\nYou have access to the following tools:\n> Tool Name: query_engine_tool\nTool Description: Useful for running a natural language query\nagainst a knowledge base and get back a natural language response.\n\nTool Args: {\'title\': \'DefaultToolFnSchema\', \'description\': \'Default tool function Schema.\', \'type\': \'object\', \'properties\': {\'input\': {\'title\': \'Input\', \'type\': \'string\'}}, \'required\': [\'input\']}\n\n\n## Output Format\nTo answer the question, please use the following format.\n\n
\nThought: I need to use a tool to help me answer the question.\nAction: tool name (one of query_engine_tool)\nAction Input: the input to the tool, in a JSON format representing the kwargs (e.g. {"text": "hello world", "num_beams": 5})\n
Plain Text
\nPlease use a valid JSON format for the action input. Do NOT do this {\'text\': \'hello world\', \'num_beams\': 5}.\n\nIf this format is used, the user will respond in the following format:\n\n
\nObservation: tool response\n
Plain Text
\n\nYou should keep repeating the above format until you have enough information\nto answer the question without using any more tools. At that point, you MUST respond\nin the following format:\n\n
\nThought: I can answer without using any more tools.\nAnswer: [your answer here]\n
Plain Text
\n\n## Current Conversation\nBelow is the current conversation consisting of interleaving human and assistant messages.\n\n<|end_of_turn|>User: Describe the Gaia character from the book \'Ranenie Zvezdy 4\' and tell the boot story short<|end_of_turn|>'
On this image I replaced \n with linefeed character, so it easier to read
Attachment
image.png
Agent ask LLM to use Thought: Answer: format. But LLM trained to use User: Assistant: <|end_of_turn|> format. Can it be the problem?
Maybe donot put Agent messages into LLM template? And try LLM to use Agent's templates
Add a reply
Sign up and join the conversation on Discord