LlamaIndex ChatEngine

jjonm

LlamaIndex ChatEngine

I'm trying to use Llamaindex without a full RAG scenario. I.e. query the LLM as a first try, but then do an internet search when it's clear that the information isn't part of it's trained data (e.g. the weather in London today). I was thinking of using both as separate query engines and then chaining them together. However, I can't yet get a simple direct chat (via LlamaIndex) to work.

The problem is that I don't need a RAG scenario. I can't see from the documentation how I might do this with LlamaIndex (I'm using a local LLM, but this should not be an issue).

Langchain in theory is better at pulling together different things without assuming a RAG use-case. Unfortunately, Langchain, is very buggy, the documentation is terrible, and out of date, and the code-base keeps changing faster than anyone can keep up with the bugs.

In short, I can think of only one thing that I like about langchain, and that is that it doesn't assume RAG.

Is it possible to do this with LlamaIndex as well? This isn't obvious (to me) in the documentation.

19 comments

LLogan M

I think you are describing an agent type of workflow right?

We support this type workflow: https://docs.llamaindex.ai/en/stable/module_guides/deploying/agents/usage_pattern.html

If you are using openai, definitely use the OpenAIAgent instead of the react agent

jjonm

Thanks @Logan M - I think it is a workflow. I.e., first check chat, and then check Google/DDG etc.

I'm using an open-source model so this might be problematic given some of the assumptions that LlamaIndex sometimes makes, but I will give it a go.

LLogan M

ooo, open-source LLMs are generally bad at agentic stuff 😅

Maybe a custom query pipeline would be more well suited here (youd have much higher control over prompts, output parsing, etc.)
https://docs.llamaindex.ai/en/stable/module_guides/querying/pipeline/root.html

jjonm

I saw that page, but one issue I have with Llamaindex docs is that there are a lot of snippets that don't seem to fit together. It's hard to see the continuity between scenarios

jjonm

I'll try to get 'something' working and then improve on at least a functional artefact

jjonm

I think that it also used a retriever, which is specifically 'docs' - i.e. RAG-based

jjonm

One of the reasons I'm using Open-source is that it's more likely to get people to trial things than be shocked by the sometimes ridiculous OpenAI bills that shouldn't surprise them, but do

jjonm

So.... I'm not seeing it flow from one tool to the next. It does do a chat, but I can't see how to set the MetaData to help differentiate between them (other than in the Google one).

Plain Text

from llama_index.query_engine import CustomQueryEngine
from llama_index.tools import BaseTool, FunctionTool

class GoogleQueryEngine(CustomQueryEngine):
    """RAG Query Engine."""
    service_context: ServiceContext
    maxhits: int=5

    def custom_query(self, query_str: str):
        googleQueryDocs = webSearchDocs(query=query_str, maxhits=self.maxhits)
        googleIndex = VectorStoreIndex.from_documents(googleQueryDocs, service_context=self.service_context)
        return googleIndex.as_query_engine(service_context=service_context).query(query_str)

google_query_engine = GoogleQueryEngine(service_context=service_context, maxhits=3)

simple_chat_engine = SimpleChatEngine.from_defaults(service_context=service_context)

def llmChat(input: str) -> str:
    response = simple_chat_engine.chat(input)
    return response

chat_tool = FunctionTool.from_defaults(fn=llmChat)

google_query_tool =  QueryEngineTool(
        query_engine=google_query_engine,
        metadata=ToolMetadata(
            name="google_agent", description="Agent that can use Google to locate content, when the data is not part of the training set"
        ),
    )

tools = [chat_tool, google_query_tool]

outer_agent = ReActAgent.from_tools(tools, llm=llm, verbose=True)

jjonm

Plain Text

response = outer_agent.chat("what time is it now in Brisbane, Australia?")

Gives me...

Plain Text

Thought: I need to use a tool to help me answer this question.
Action: llmChat
Action Input: {'input': 'what time is it now in Brisbane, Australia?'}
Observation: <|>
I do not have access to real-time information. however, you can check the current time in brisbane, australia by visiting a reliable time zone converter website or using your device's clock feature and setting it to brisbane's time zone (utc+10).
Thought: I can answer without using any more tools.
Answer: The current time in Brisbane, Australia is [insert current time in Brisbane, Australia].

jjonm

It looks like it uses the LLM, makes up some stuff, and then decides not to use the search capability, even though there is metadata to tell it to use it.

jjonm

hahaha... I changed the prompt slightly

Plain Text

response = outer_agent.chat("what time is it now in Brisbane, Australia?. Show your working, and validate the results")

And this is what it came back with...

Plain Text

Thought: I need to use a tool to help me answer this question.
Action: llmChat(input="what time is it now in Brisbane, Australia?")
Action Input: {"input": "what time is it now in Brisbane, Australia?", "num_beams": 1}

Observation: The current time in Brisbane, Australia is [insert current time in Brisbane, Australia].
Thought: I can answer without using any more tools.
Answer: The current time in Brisbane, Australia is [insert current time in Brisbane, Australia]. To validate the results, you can use a tool like google_agent to search for the current time in Brisbane, Australia and compare it with my response.

LLogan M

haha yeaaa -- it can be quite hard to steer some LLMs to actually pick and use a tool in my experience. They really like to hallucinate :PSadge:

What LLM are you using? Some are certainly better than others

jjonm

Zephyr Alpha 7B

jjonm

It looks like part of the problem is that the standard BeautifulSoupWebReader is returning very poor content with a challenging web page. The LLM can only do so much. I've found an alternative approach, so will try to create an alternative BeautifulSoupReader

jjonm

@Logan M Yes!! It looks like the LlamaIndex BeautifulSoupWebReader is not great, and therefore the LLM had very little to work with. I've created an alternative one that gave me the exact time from a google search. I'll do a PR

jjonm

It simply reduces the content by filtering on relevant tags

jjonm

PR created

LLogan M

awesome! thanks!

jjonm

Plain Text

Thought: I need to use a tool to help me answer the question.
Action: llmChat
Action Input: {'input': 'Using an internet search, tellme what time is it now in Sydney, Australia?. Show your working, and validate the results'}
Observation: I do not have access to real-time data or the ability to browse the internet. however, i can provide you with the current time in sydney, australia based on my database. as of my last update, which was a few seconds ago, it is currently 10:25 pm (australian eastern standard time) in sydney, australia. to validate this result, you can check any reliable time zone converter or the official website of the sydney observatory.

Thought: I need to use a tool to help me answer the question.
Action: google_agent
Action Input: {'input': 'Using an internet search, tellme what time is it now in Sydney, Australia?. Show your working, and validate the results'}
Searching for relevant documents in Google
Observation: 
I used Google to search for "what time is it now in sydney australia?" The first result that appeared was from TimeandDate.com, which showed me that as of writing this answer (10:35 AM on February 9th, 2024), the current time in Sydney, Australia is 02:04 PM.

To validate the results, I checked the same information on TimeandDate.com's website. The website shows that as of writing this answer (10:35 AM on February 9th, 2024), the current time in Sydney, Australia is 02:04 PM.

Add a reply

Find answers from the community

LlamaIndex ChatEngine