Hi, my React agent is initialized with

aathenawisdoms

Hi, my React agent is initialized with some tools, but when it decides to use a FunctionTool, the output appears to be hallucinated.

Plain Text

Thought: The user has provided his preference. I can use the select_game tool to find the appropriate game for him.
Action: select_game
Action Input: {"preference_description": "racing"}

Observation: {"game_name": "Forza Horizon"}

The function select_game will never give the above ToolOutput.
Shouldnt it have the actual output of the Functiontool.fn as it's Observation?

Plain Text

tools = []

def select_game(preference_description: str) -> Dict[str, str]:
    """Use the user's description of his preference to select a game for him."""
    return random.choice([{
        "game_name": "Final Fantasy",
        "webpage": "http://ff7.game",
    }, {
        "game_name": "Call of Duty",
        "webpage": "http://cod.game",
    }])

tools.append(FunctionTool.from_defaults(fn=select_game))

agent = ReActAgent.from_tools(tools, llm=llm, verbose=True)

Also it seems like there's an unexpected newline between Action Input and Observation, so maybe this causes the react steps to be part of the agent response...

6 comments

LLogan M

I think your agent hallucinated the entire loop 👀

LLogan M

(hence the weird spacing and odd output)

aathenawisdoms

Oh thats not good... I am using mixtral-8x7b-instruct unquantized.
How do I know if its actually calling the tools or hallucinating the loop?

LLogan M

The prints should hopefully be colored properly (each color is one output)

So here, theres pink (initial call) blue (the tool output) and pink again (the final response)

Attachment

VVicent W.

@athenawisdoms , a dirty trick I figured was adding Observation: as a stop token to your local LLM server, like this:

https://github.com/tslmy/agent/blob/e330255806c97a93a733dab3edd9a843902375f5/main.py#L60

Plain Text

    Settings.llm = Ollama(
        model="zephyr:7b-beta",
        timeout=600,  # secs
        streaming=True,
        callback_manager=callback_manager,
        additional_kwargs={"stop": ["Observation:"]},
    )

This way, when LLM is about to say "Observation:", your API server will say "hol' up" and cut it off there.

VVicent W.

It has greatly mitigated hallucination for my Zephyr 7B beta LLM. Give it a try!

Add a reply

Find answers from the community

Hi, my React agent is initialized with