I have not come across any yet π€
Yea every open source model I've tried sucks at being an agent lol even zephyr
It could be a matter of prompt engineering, but that's also annoying
cool, will check it out.
The challenge I'm finding with ReAact is more in Langchain's specific implementation of the ReAct agent, and, more specifically, the need for the first "Thought" prompt, even though it's empty.
tl;dr it has to do with python templating and the intermediary thoughts/agent_scratchpad implementation
I'm able to get many of the models to generate the first thought when the incomplete thought prompt is absent
but introducing the incomplete thought prompt throws them all off
Was our react agent doing the same thing?
(I know barely anything about langchains version lol)
i haven't tried your react agent
does your agent expect to use the RAG?
oh, funny, someone at RH sent me THUDM/agentlm which I just tried - it also fails at react (in the 13b variant)
they all fail to follow the prompt
I think at least with that dataset, you can finetune an LLM with the dataset for react agents right?
it looks like it would involve literally re-writing the datasets in ReAct format, I think
They are already in a react format, albeit something a little different compared to llamaindex or langchain
let me double check. it didn't look like agentinstruct was in react format
ok yeah it does look like react
i don't think this is a fine-tuning problem though
Answer the following questions as best you can. You have access to the following tools:
Search: A search engine. Useful for when you need to answer questions about current events. Input should be a search query.
Calculator: Useful for when you need to answer questions about math.
Use the following format:
Thought: you should always think about what to do
Action: the action to take, should be one of [Search, Calculator]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
Question: Who was president when John F. Kennedy was in middle school?
Thought:
this is vanilla react from langchain
langchain expects that the model will re-generate Thought
buried in langchain is a PromptTemplate
that tries to inject the agent's temporary thoughts into that thought, so if you don't include the value for the temp memory, langchain blows up
need to try llamaindex here
your react agent expects json input, FWIW
i'm trying to debug as usual and struggling to get llamaindex to output stuff in a way I can understand
which part of the following is actually sent to the model?
Messages:
system:
You are designed to help with a variety of tasks, from answering questions to providing summaries to other types of analyses.
Tools
You have access to a wide variety of tools. You are responsible for using
the tools in any sequence you deem appropriate to complete the task at hand.
This may require breaking the task into subtasks and using different tools
to complete each subtask.
You have access to the following tools:
> Tool Name: multiply
Tool Description: multiply(a: int, b: int) -> int
Multiple two integers and returns the result integer
Tool Args: {'title': 'multiply', 'type': 'object', 'properties': {'a': {'title': 'A', 'type': 'integer'}, 'b': {'title': 'B', 'type': 'integer'}}, 'required': ['a', 'b']}
Output Format
To answer the question, please use the following format.
Thought: I need to use a tool to help me answer the question.
Action: tool name (one of multiply)
Action Input: the input to the tool, in a JSON format representing the kwargs (e.g. {"text": "hello world", "num_beams": 5})
Please use a valid JSON format for the action input. Do NOT do this {'text': 'hello world', 'num_beams': 5}.
If this format is used, the user will respond in the following format:
Observation: tool response
You should keep repeating the above format until you have enough information
to answer the question without using any more tools. At that point, you MUST respond
in the following format:
Thought: I can answer without using any more tools.
Answer: [your answer here]
Current Conversation
Below is the current conversation consisting of interleaving human and assistant messages.
user: What is 2123 * 215123
**
Response:
assistant: ?
**
seems like llama2-70b-chat and falcon-180b can deal with your react agent OK
i still think asking models like this to generate structured JSON is fragile. but it seems to work
That entire blob is the prompt. React is hard
It's either json parsing or regex parsing, pick your poison π€·ββοΈ lol
This aligns with my experience. React is tough for small models these days
I don't know that you have to do much in the way of regex parsing
the list of tools has names. you ask the model to name the tool in Action:
and then the input to the tool in Action Input:
which is how Langchain does it
they're just too rigid in the expectation of the model generating a Thought:
as the first part of the response