Find answers from the community

Updated 2 months ago

LlamaIndex ChatEngine

LlamaIndex ChatEngine

I'm trying to use Llamaindex without a full RAG scenario. I.e. query the LLM as a first try, but then do an internet search when it's clear that the information isn't part of it's trained data (e.g. the weather in London today). I was thinking of using both as separate query engines and then chaining them together. However, I can't yet get a simple direct chat (via LlamaIndex) to work.

The problem is that I don't need a RAG scenario. I can't see from the documentation how I might do this with LlamaIndex (I'm using a local LLM, but this should not be an issue).

Langchain in theory is better at pulling together different things without assuming a RAG use-case. Unfortunately, Langchain, is very buggy, the documentation is terrible, and out of date, and the code-base keeps changing faster than anyone can keep up with the bugs.

In short, I can think of only one thing that I like about langchain, and that is that it doesn't assume RAG.

Is it possible to do this with LlamaIndex as well? This isn't obvious (to me) in the documentation.
L
j
19 comments
I think you are describing an agent type of workflow right?

We support this type workflow: https://docs.llamaindex.ai/en/stable/module_guides/deploying/agents/usage_pattern.html

If you are using openai, definitely use the OpenAIAgent instead of the react agent
Thanks @Logan M - I think it is a workflow. I.e., first check chat, and then check Google/DDG etc.

I'm using an open-source model so this might be problematic given some of the assumptions that LlamaIndex sometimes makes, but I will give it a go.
ooo, open-source LLMs are generally bad at agentic stuff πŸ˜…

Maybe a custom query pipeline would be more well suited here (youd have much higher control over prompts, output parsing, etc.)
https://docs.llamaindex.ai/en/stable/module_guides/querying/pipeline/root.html
I saw that page, but one issue I have with Llamaindex docs is that there are a lot of snippets that don't seem to fit together. It's hard to see the continuity between scenarios
I'll try to get 'something' working and then improve on at least a functional artefact
I think that it also used a retriever, which is specifically 'docs' - i.e. RAG-based
One of the reasons I'm using Open-source is that it's more likely to get people to trial things than be shocked by the sometimes ridiculous OpenAI bills that shouldn't surprise them, but do
So.... I'm not seeing it flow from one tool to the next. It does do a chat, but I can't see how to set the MetaData to help differentiate between them (other than in the Google one).

Plain Text
from llama_index.query_engine import CustomQueryEngine
from llama_index.tools import BaseTool, FunctionTool

class GoogleQueryEngine(CustomQueryEngine):
    """RAG Query Engine."""
    service_context: ServiceContext
    maxhits: int=5

    def custom_query(self, query_str: str):
        googleQueryDocs = webSearchDocs(query=query_str, maxhits=self.maxhits)
        googleIndex = VectorStoreIndex.from_documents(googleQueryDocs, service_context=self.service_context)
        return googleIndex.as_query_engine(service_context=service_context).query(query_str)

google_query_engine = GoogleQueryEngine(service_context=service_context, maxhits=3)

simple_chat_engine = SimpleChatEngine.from_defaults(service_context=service_context)

def llmChat(input: str) -> str:
    response = simple_chat_engine.chat(input)
    return response

chat_tool = FunctionTool.from_defaults(fn=llmChat)

google_query_tool =  QueryEngineTool(
        query_engine=google_query_engine,
        metadata=ToolMetadata(
            name="google_agent", description="Agent that can use Google to locate content, when the data is not part of the training set"
        ),
    )

tools = [chat_tool, google_query_tool]

outer_agent = ReActAgent.from_tools(tools, llm=llm, verbose=True)
Plain Text
response = outer_agent.chat("what time is it now in Brisbane, Australia?")


Gives me...

Plain Text
Thought: I need to use a tool to help me answer this question.
Action: llmChat
Action Input: {'input': 'what time is it now in Brisbane, Australia?'}
Observation: <|>
I do not have access to real-time information. however, you can check the current time in brisbane, australia by visiting a reliable time zone converter website or using your device's clock feature and setting it to brisbane's time zone (utc+10).
Thought: I can answer without using any more tools.
Answer: The current time in Brisbane, Australia is [insert current time in Brisbane, Australia].
It looks like it uses the LLM, makes up some stuff, and then decides not to use the search capability, even though there is metadata to tell it to use it.
hahaha... I changed the prompt slightly

Plain Text
response = outer_agent.chat("what time is it now in Brisbane, Australia?. Show your working, and validate the results")


And this is what it came back with...

Plain Text
Thought: I need to use a tool to help me answer this question.
Action: llmChat(input="what time is it now in Brisbane, Australia?")
Action Input: {"input": "what time is it now in Brisbane, Australia?", "num_beams": 1}

Observation: The current time in Brisbane, Australia is [insert current time in Brisbane, Australia].
Thought: I can answer without using any more tools.
Answer: The current time in Brisbane, Australia is [insert current time in Brisbane, Australia]. To validate the results, you can use a tool like google_agent to search for the current time in Brisbane, Australia and compare it with my response.
haha yeaaa -- it can be quite hard to steer some LLMs to actually pick and use a tool in my experience. They really like to hallucinate :PSadge:

What LLM are you using? Some are certainly better than others
Zephyr Alpha 7B
It looks like part of the problem is that the standard BeautifulSoupWebReader is returning very poor content with a challenging web page. The LLM can only do so much. I've found an alternative approach, so will try to create an alternative BeautifulSoupReader
@Logan M Yes!! It looks like the LlamaIndex BeautifulSoupWebReader is not great, and therefore the LLM had very little to work with. I've created an alternative one that gave me the exact time from a google search. I'll do a PR
It simply reduces the content by filtering on relevant tags
awesome! thanks!
Plain Text
Thought: I need to use a tool to help me answer the question.
Action: llmChat
Action Input: {'input': 'Using an internet search, tellme what time is it now in Sydney, Australia?. Show your working, and validate the results'}
Observation: I do not have access to real-time data or the ability to browse the internet. however, i can provide you with the current time in sydney, australia based on my database. as of my last update, which was a few seconds ago, it is currently 10:25 pm (australian eastern standard time) in sydney, australia. to validate this result, you can check any reliable time zone converter or the official website of the sydney observatory.

Thought: I need to use a tool to help me answer the question.
Action: google_agent
Action Input: {'input': 'Using an internet search, tellme what time is it now in Sydney, Australia?. Show your working, and validate the results'}
Searching for relevant documents in Google
Observation: 
I used Google to search for "what time is it now in sydney australia?" The first result that appeared was from TimeandDate.com, which showed me that as of writing this answer (10:35 AM on February 9th, 2024), the current time in Sydney, Australia is 02:04 PM.

To validate the results, I checked the same information on TimeandDate.com's website. The website shows that as of writing this answer (10:35 AM on February 9th, 2024), the current time in Sydney, Australia is 02:04 PM.
Add a reply
Sign up and join the conversation on Discord