Anyone knows why instantiating the SAME

At a glance

The community member is experiencing an issue where a Bedrock LLM agent works well in a Gradio app, but when used in a FastAPI app created by the create-llama command line tool, the agent is unable to call the tools and instead produces hallucinations. The issue only occurs when using the Bedrock agent, as the agent works fine with OpenAI but not with other models like Claude. The Bedrock agent is able to find the tools, but it is unable to receive the results when outputting the Observation, leading to hallucinations. This behavior is inconsistent with the Gradio app where the agent works as expected.

In the comments, another community member is unsure if the create-llama FastAPI app is even using an agent, but a second community member confirms that if the "just a simple chatbot or agent" option is chosen, an AgentRunner instance is created with a selection of tools like Wikipedia. The second community member also tried using the same Bedrock agent code in the create-llama app, but it always fails when using the tools.

BBenjamin Bascary

Anyone knows why instantiating the SAME Bedrock llm agent in gradio works great and is able to call the tools, but when using the agent with the fastapi app provided by create-llama command line tool it is not able to call the tools, it just allucinates. It only works with OpenAI, but when switching to claude (for example) it is not able to use the tools. The Bedrock agent is able to find the tools but it is not able to receive the results when outputing the Observation, its just hallucinations. Its weird because in the gradio app it just works.

create-llama:


@r.post("/request")
async def chat_request(
    data: _ChatData,
    chat_engine: BaseChatEngine = Depends(get_chat_engine),
):
    last_message_content, messages = await parse_chat_data(data)

    response = chat_engine.chat(last_message_content, messages)
    return _Result(
        result=_Message(role=MessageRole.ASSISTANT, content=response.response),
        nodes=_SourceNodes.from_source_nodes(response.source_nodes),
    )

Gradio app:

def run_agent(query: str) -> str:
    prompt = ...

    tool_names = [tool.metadata.name for tool in tools]

    tool_names_str = "\n".join(tool_names)

    qa_template = prompt.format(query=query, tools=tool_names_str)

    llm = Bedrock(
        model=os.getenv("MODEL_ID"),
        aws_access_key_id=os.getenv("BEDROCK_AWS_ACCESS_KEY"),
        aws_secret_access_key=os.getenv("BEDROCK_AWS_SECRET_KEY"),
        system_prompt=qa_template,
        temperature=0,
        region_name="us-east-1",
    )

    agent = ReActAgent.from_tools(tools, llm=llm, verbose=True, max_iterations=40)

    response = agent.chat(qa_template)

    return response

theme = gr.themes.Default(
    primary_hue="amber",
)

2 comments

LLogan M

I'm not sure if this is comparable -- is create-llama even using an agent?

BBenjamin Bascary

Yes, if you choose fastapi backend and no data and go with the "just a simple chatbot or agent" it creates an agent ( AgentRunner instance) with a choose of tools like Wikipedia and others, I tried bringing a Bedrock agent code (identical) to the create-llama app and it always fails when using the tools

Add a reply

Find answers from the community

Anyone knows why instantiating the SAME