Anyone knows why instantiating the SAME

Anyone knows why instantiating the SAME Bedrock llm agent in gradio works great and is able to call the tools, but when using the agent with the fastapi app provided by create-llama command line tool it is not able to call the tools, it just allucinates. It only works with OpenAI, but when switching to claude (for example) it is not able to use the tools. The Bedrock agent is able to find the tools but it is not able to receive the results when outputing the Observation, its just hallucinations. Its weird because in the gradio app it just works.

create-llama:


@r.post("/request")
async def chat_request(
    data: _ChatData,
    chat_engine: BaseChatEngine = Depends(get_chat_engine),
):
    last_message_content, messages = await parse_chat_data(data)

    response = chat_engine.chat(last_message_content, messages)
    return _Result(
        result=_Message(role=MessageRole.ASSISTANT, content=response.response),
        nodes=_SourceNodes.from_source_nodes(response.source_nodes),
    )

Gradio app:

def run_agent(query: str) -> str:
    prompt = ...

    tool_names = [tool.metadata.name for tool in tools]

    tool_names_str = "\n".join(tool_names)

    qa_template = prompt.format(query=query, tools=tool_names_str)

    llm = Bedrock(
        model=os.getenv("MODEL_ID"),
        aws_access_key_id=os.getenv("BEDROCK_AWS_ACCESS_KEY"),
        aws_secret_access_key=os.getenv("BEDROCK_AWS_SECRET_KEY"),
        system_prompt=qa_template,
        temperature=0,
        region_name="us-east-1",
    )

    agent = ReActAgent.from_tools(tools, llm=llm, verbose=True, max_iterations=40)

    response = agent.chat(qa_template)

    return response

theme = gr.themes.Default(
    primary_hue="amber",
)

Find answers from the community

Anyone knows why instantiating the SAME