Find answers from the community

Updated 3 months ago

LIRag - Pastebin.com

llama index always wants to use openai, despite me specifying not to use it in my app. I'm assuming that I am calling for the models incorrectly. Can somebody look at my code and let me know what I'm doing wrong: https://pastebin.com/9ddUR9mb as of now the only way I can get it to work is by modifying both llms/utils.py and embeddings/utils.py within the llama_index module

5 comments

LLogan M

At the top of your code, you are either loading or creating a new index without specifying a sevrice context in either (both from_documents() and load_index_from_storage() need the service context as a kwarg)

LLogan M

you can probably just get away with a global service context here at the top

Plain Text

from llama_index import set_global_service_context

set_global_service_context(service_context)

DDrewzy

Oi that makes perfect sense lol. Thanks for helping me gain a deeper understanding. Unfortunately, while making those changes did solve my original issue, it's made the app considerably less performant. I'm going to hit the docs and possible build back up from scratch with the knowledge I've gained thus far.

DDrewzy

Thanks again (as always) Logan!

LLogan M

Yea running local models is both hard and usually slower 😅 But there are hosting options like vLLM or text-generation-interface to help speed things up. Not sure if ollama has any tricks for this as well.

Add a reply