Using Ollama - Instructor

At a glance

The community member is interested in using the Ollama llama3 model with their PydanticProgram and wants to reduce costs by moving to local models. Another community member provides an example of how to use Ollama with Pydantic, including setting json_mode=True. However, another community member encounters an issue with the FunctionCallingAgentWorker and is advised to use a ReActAgentWorker instead, as Ollama does not have a tool calling API. The community members also discuss increasing the timeout and using additional_kwargs={"stop": ["Observation:"]} to help with output parsing. In the end, the issue is resolved, and the community member reports that the solution worked.

Useful resources

NNehil

Is it possible to get PydanticProgram working with Ollama llama3? With instructor its super easy to do https://python.useinstructor.com/hub/ollama/#patching/. I have a loading pipeline with uses pydanticextractor. I want to reduce cost by moving to local models.

12 comments

LLogan M

Pretty easy to do

Plain Text

from llama_index.llms.ollama import Ollama
from llama_index.core.prompts import PromptTemplate
from pydantic.v1 import BaseModel, Field

class MyClass(BaseModel):
  """Some description."""
  name: str = Field(description="Some description")

llm = Ollama(..., json_mode=True)

prompt = PromptTemplate("Give me a name based on {topic}")
output = llm.structured_predict(MyClass, prompt, topic="movies")
print(output.name)

# or async
output = await llm.astructured_predict(MyClass, prompt, topic="movies")

AAashiDutt

Hey @Nehil, I am trying to implement the same with Ollama but with tool calling. But I got the following error

Cell In[32], line 6
2 from llama_index.core.agent import FunctionCallingAgentWorker
3 from llama_index.core.agent import AgentRunner
----> 6 agent_worker = FunctionCallingAgentWorker.from_tools(
7 initial_tools,
8 llm = llm,
9 verbose = True
10 )
13 agent = AgentRunner(agent_worker)

File ~/miniconda3/envs/DL/lib/python3.10/site-packages/llama_index/core/agent/function_calling/step.py:125, in FunctionCallingAgentWorker.from_tools(cls, tools, tool_retriever, llm, verbose, max_function_calls, callback_manager, system_prompt, prefix_messages, kwargs) 121 prefix_messages = [ChatMessage(content=system_prompt, role="system")] 123 prefix_messages = prefix_messages or []--> 125 return cls( 126 tools=tools, 127 tool_retriever=tool_retriever, 128 llm=llm, 129 prefix_messages=prefix_messages, 130 verbose=verbose, 131 max_function_calls=max_function_calls, 132 callback_manager=callback_manager, 133 kwargs,
134 )
...
71 )
72 self._llm = llm
73 self._verbose = verbose

ValueError: Model name mistral does not support function calling API.

Could you please help me?

LLogan M

@AashiDutt Ollama doesn't have a tool calling API. You'll have to use a ReActAgentWorker or something else

AAashiDutt

Could you provide some link or reference for this?

LLogan M

Its the exact same, just a different import

Plain Text

from llama_index.core.agent import ReActAgentWorker

agent_worker = ReActAgentWorker.from_tools(initial_tools, llm=llm, verbose=True)

AAashiDutt

Got it. Thank You 🙂

LLogan M

You might find it helpful with ollama to use Ollama(,,,, json_mode=True), or alternatively setting Ollama(..., additional_kwargs={"stop": ["Observation:"]}) to help with output parsing

AAashiDutt

I did use json_mode = True, but I'm encountering ReadTimeout: timed out.

LLogan M

You can increase the timeout

LLogan M

Ollama(... request_timeout=3600)

LLogan M

Should be enough lol

AAashiDutt

It worked ! 🤩

Add a reply

Find answers from the community

Using Ollama - Instructor