```

At a glance

The community members are discussing an issue with running the "dolphin-phi" application on a 5600g AMD APU using the Ollama library. The main points are:

- The issue seems to be related to an HTTP request timeout, and the community members suggest increasing the default request_timeout for Ollama in the constructor.

- They also discuss the version of llama-index being used, and note that version 0.9.22 added the request_timeout parameter.

- The community members try different approaches, such as changing the base.py file and using the Mistral API instead of Ollama, in an attempt to resolve the issue.

There is no explicitly marked answer in the comments.

Useful resources

mmr.niko.la

Plain Text

is this happen when you run out of memory?

im running dolphin-phi on 5600g amd apu using ollama

21 comments

LLogan M

I think it's just an http request timeout

LLogan M

You can increase the default request_timeout for ollama in the constructor

LLogan M

default is 30s

mmr.niko.la

this is ollama or llama index limit?

LLogan M

llama-index

LLogan M

Basically need to add llm = Ollama(..., request_timeout=60) or some other timeout value

mmr.niko.la

llm = Ollama(model=self._model, base_url=self._base_url, request_timeout=300)

strange changed the base.py in the OllamaQueryEnginePack

stll time out in 30s

when rerunning

ollama_pack = OllamaQueryEnginePack(model="dolphin-phi", documents=documents)
response = ollama_pack.run("What is the title of the doc?")

LLogan M

What version of llama-index do you have?

LLogan M

v0.9.22 added the request_timeout kwarg

LLogan M

Attachments

mmr.niko.la

Plain Text

WARNING: Package(s) not found: langchain
Name: llama-index
Version: 0.9.25.post1
Name: transformers
Version: 4.36.2
Name: sentence-transformers
Version: 2.2.2
Name: pypdf
Version: 3.17.4
Note: you may need to restart the kernel to use updated packages.

python 3.11.5

Attachment

LLogan M

I'm at a loss for this one. As you can see in the traceback, the timeout is being passed in

Plain Text

    101 with httpx.Client(timeout=Timeout(self.request_timeout)) as client:
--> 102     response = client.post(
    103         url=f"{self.base_url}/api/chat",
    104         json=payload,
    105     )

We are even following httpx docs
https://www.python-httpx.org/advanced/#setting-a-default-timeout-on-a-client

LLogan M

This seems to require some rigorous debugging :PSadge:

mmr.niko.la

Hmmm interesting 🧐

mmr.niko.la

in the base.py can we comment out
llm = Ollama(model=self._model, base_url=self._base_url)

instead add

llm = MistralAI(api_key=api_key)

im moving to Mistral api to still built the rag while still debuging the timeout

mmr.niko.la

if not do you know of any working rag pdf for any mistral cloud api demo. i need to get back on track and llocal llm with ollama is bugging out for the last week.

LLogan M

it should be as simple as swapping out the llm 👀

mmr.niko.la

Plain Text

so ollama serve runs on boot as it was installed from their official script

when running the dolphin-phi its 8GB/16gb

it was working pretty good on m1 mac mini 8gb. lmfao fml

Attachment

mmr.niko.la

@Logan M lmk if you see anything out of order.

LLogan M

whats the issue? 👀

LLogan M

(I feel blind lol but that looks fine to me?)

Add a reply

Find answers from the community

```