Hi I am trying to create a Retrieval

eelmatero

Hi, I am trying to create a Retrieval-augmented openAi agent (https://gpt-index.readthedocs.io/en/stable/examples/agent/openai_agent_retrieval.html#retrieval-augmented-openai-agent)
I used to use an OpenAIAgent. I don't understand how can use a query_engine_tool. I am recieving an error: Error: Tool with name query_engine_tool not found
I am using the FnRetrieverOpenAIAgent wrong and I don't know how to use it right. I couldn't find anything in the docs. This is my actual code:

self._user_id = user_id
self.project_id = index_dir
self._tools = {tool.metadata.name: tool for tool in tools}
_vector_store = FaissVectorStore.from_persist_dir(
    f"./llama_index/{index_dir}"
)
_storage_context = StorageContext.from_defaults(
    vector_store=_vector_store,
    persist_dir=f"./llama_index/{index_dir}",
)
_index = load_index_from_storage(storage_context=_storage_context)
_memory = ChatMemoryBuffer.from_defaults(
    token_limit=3000, chat_history=chat_history
)
similarity_top_k = 7
_retriever = VectorIndexRetriever(
    index=_index, similarity_top_k=similarity_top_k
)
_query_engine = RetrieverQueryEngine(retriever=_retriever)
query_engine_tool = QueryEngineTool.from_defaults(
    query_engine=_query_engine,
)
_all_tools = [query_engine_tool]
for tool in tools:
    _all_tools.append(tool)
tool_mapping = SimpleToolNodeMapping.from_objects(_all_tools)
obj_index = ObjectIndex.from_objects(
    _all_tools,
    tool_mapping,
    VectorStoreIndex,
)
self.token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode,
)
callback_manager = CallbackManager([self.token_counter])
self._agent = FnRetrieverOpenAIAgent.from_retriever(
    retriever=obj_index.as_retriever(),
    llm=llm,
    callback_manager=callback_manager,
    memory=_memory,
    system_prompt=TEXT_QA_SYSTEM_PROMPT.content,
    verbose=True,
)
self._chat_history = chat_history

24 comments

LLogan M

well, since you only have one tool, do don't need to use this.

But in any case, you should give the tool a name and description

Plain Text

query_engine_tool = QueryEngineTool.from_defaults(
    query_engine=_query_engine,
    name="query_engine",
    description="Useful for asking questions about X"
)

eelmatero

yess, this are the tools that I pass:
tools: Sequence[BaseTool] = [
imagenes_tool,
electro_tool,
gamma_tool,
endoscopia_tool,
laboratorio_tool,
historias_clinicas_tool,
vacunatorio_tool,
sangre_tool,
useless_empresa_tool,
cobertura_medica_tool,
urgencia_tool,
oftalmologia_cerro_tool,
oftalmologia_nueva_tool,
crear_cuenta_tool,
especialidades_tool,
useless_paciente_tool,
useless_derechos_paciente_tool,
useless_internacion_tool,
useless_internacional_tool,
useless_preventivo_tool,
useless_covid_tool,
useless_consejos_tool,
coseguro_tool,
useless_telemedicina_tool,
ginecologia_tool,
cancelaciones_profesional_tool,
reclamos_tool,
useless_mision_vision_valores_tool,
useless_autoridades_sanatorio_tool,
useless_infraestructura_sanatorio_tool,
useless_fundacion_allende_tool,
empleos_sanatorio_tool,
useless_calidad_sanitaria_tool,
useless_bioetica_tool,
useless_capacitacion_docencia_tool,
useless_investigaciones_tool,
useless_control_infecciones_tool,
useless_comites_cientificos_tool,
useless_enfermeria_tool,
useless_trabajos_cientificos_tool,
useless_actividades_profesionales_tool,
useless_proveedores_sanatorio_tool,
],

LLogan M

make sure you gave each tool an actualy name and description then I think.

If you are, what's the full traceback you are getting?

eelmatero

this is the full traceback:

Error: Tool with name python not found
Traceback (most recent call last):
  File "/home/project/venv/lib/python3.9/site-packages/gradio/queueing.py", line 388, in call_prediction
    output = await route_utils.call_process_api(
  File "/home/project/venv/lib/python3.9/site-packages/gradio/route_utils.py", line 219, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/project/venv/lib/python3.9/site-packages/gradio/blocks.py", line 1437, in process_api
    result = await self.call_function(
  File "/home/project/venv/lib/python3.9/site-packages/gradio/blocks.py", line 1107, in call_function
    prediction = await fn(*processed_input)
  File "/home/project/venv/lib/python3.9/site-packages/gradio/utils.py", line 616, in async_wrapper
    response = await f(*args, **kwargs)
  File "/home/project/testing_gradio.py", line 9, in chatbot_response
    return response["content"]
TypeError: 'NoneType' object is not subscriptable

I changed this code like this:

        _all_tools = [query_engine_tool]
        for tool in tools:
            _all_tools.append(tool)
        print(_all_tools[0].metadata)

And I got:

ToolMetadata(description='Se utiliza para obtener información sobre los médicos', name='query_engine_tool', fn_schema=<class 'llama_index.tools.types.DefaultToolFnSchema'>)

The FunctionTool Objects works fine, the problem is only the QueryEngineTool

eelmatero

that traceback is more for the gradio, I get the principal error with this code

        except Exception as e:
            print(f"Error: {e}")

eelmatero

I think that the error is here:

        obj_index = ObjectIndex.from_objects(
            _all_tools,
            tool_mapping,
            VectorStoreIndex,
        )

I think that the VectorStoreIndex has to be other object from my code. But I don't know how to integrate it

eelmatero

If I print the chat history I get this:

ChatMessage(role=<MessageRole.USER: 'user'>, content='y que medicos pediatras tienen?', additional_kwargs={}), ChatMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=None, additional_kwargs={'function_call': <OpenAIObject at 0x7fb75ae6a630> JSON: {
  "name": "query_engine_tool",
  "arguments": "{\"specialty\": \"Pediatr\u00eda\"}"
}})

eelmatero

as the query_engine_tool was called

LLogan M

What happens if you do obj_index.as_retriever().retrieve("y que medicos pediatras tienen?") ? Does it return a list of query engines?

eelmatero

yess, printing this:

print(
            "here: ",
            obj_index.as_retriever().retrieve("y que medicos pediatras tienen?"),
        )

I get:

here:  [<llama_index.tools.function_tool.FunctionTool object at 0x7fb181a6c130>, <llama_index.tools.function_tool.FunctionTool object at 0x7fb181a71c70>]

And if I print the query engine tool i get:
<llama_index.tools.query_engine.QueryEngineTool object at 0x7fb69c9e7be0>

eelmatero

I didn't get a list of queryes engines, I get a list of FunctionsTools

LLogan M

right thats fine, that's what it should be I think

LLogan M

Not sure what to tell you -- the example notebook ran fine for me lol

LLogan M

I'm not noticing the mistake here 🤔

eelmatero

the example uses all functions callings

eelmatero

I want to use an own index too

eelmatero

it's posible??

LLogan M

it is

eelmatero

but If I asked y que medicos pediatras tienen? why doesn't brings the QueryEngineTool function??

eelmatero

Can I configurate to get more functions than 2??

eelmatero

or to bring always the query_engine_too?

eelmatero

I got it.
I have some lines in the system_prompt asking for the query_engine_tool. And this only brings two tools. So I want to always brint this tool that is elemental

eelmatero

I can do it with this no??
response = self._agent.chat(input_text, function_call="query_engine_tool")

eelmatero

that is no working jeje. Any ideas @Logan M ?

Add a reply

Find answers from the community

Hi I am trying to create a Retrieval