Find answers from the community

Updated 7 months ago

Anyone knows why instantiating the SAME

Anyone knows why instantiating the SAME Bedrock llm agent in gradio works great and is able to call the tools, but when using the agent with the fastapi app provided by create-llama command line tool it is not able to call the tools, it just allucinates. It only works with OpenAI, but when switching to claude (for example) it is not able to use the tools. The Bedrock agent is able to find the tools but it is not able to receive the results when outputing the Observation, its just hallucinations. Its weird because in the gradio app it just works.

create-llama:
@r.post("/request") async def chat_request( data: _ChatData, chat_engine: BaseChatEngine = Depends(get_chat_engine), ): last_message_content, messages = await parse_chat_data(data) response = chat_engine.chat(last_message_content, messages) return _Result( result=_Message(role=MessageRole.ASSISTANT, content=response.response), nodes=_SourceNodes.from_source_nodes(response.source_nodes), )

Gradio app:

def run_agent(query: str) -> str: prompt = ... tool_names = [tool.metadata.name for tool in tools] tool_names_str = "\n".join(tool_names) qa_template = prompt.format(query=query, tools=tool_names_str) llm = Bedrock( model=os.getenv("MODEL_ID"), aws_access_key_id=os.getenv("BEDROCK_AWS_ACCESS_KEY"), aws_secret_access_key=os.getenv("BEDROCK_AWS_SECRET_KEY"), system_prompt=qa_template, temperature=0, region_name="us-east-1", ) agent = ReActAgent.from_tools(tools, llm=llm, verbose=True, max_iterations=40) response = agent.chat(qa_template) return response theme = gr.themes.Default( primary_hue="amber", )
L
B
2 comments
I'm not sure if this is comparable -- is create-llama even using an agent?
Yes, if you choose fastapi backend and no data and go with the "just a simple chatbot or agent" it creates an agent ( AgentRunner instance) with a choose of tools like Wikipedia and others, I tried bringing a Bedrock agent code (identical) to the create-llama app and it always fails when using the tools
Add a reply
Sign up and join the conversation on Discord