having an issue migrating to 0.10, for

mmaybe goats dont exist

having an issue migrating to 0.10, for some reason, even though im passing the embeding model to my query engines and retrievers, its still complaining i dont have an open_ai key set, as its trying to use that as a default

This is ok because im not using openAI, but is there a better way to set this up? I saw something about settings but not sure. Quite annoying!

21 comments

LLogan M

Set the global default, this is the way 🙏

LLogan M

Plain Text

from llama_index.core import Settings

Settings.llm = ...
Settings.embed_model = ...

rrobwhelanatx

does this replace set_global_service_context , a method I cannot find anymore?

mmaybe goats dont exist

The issue is that these change per chat, so if a user wants to chat with a different model I don't really want to update global settings

LLogan M

That method is still there (from llama_index.core import set_global_service_context). And under the hood that method does the above

LLogan M

Then you need to rely on passing into the interfaces

For example

Plain Text

index = VectorStoreIndex.from_documents(..., embed_model=embed_model)

index.as_chat_engine(..., llm=llm)

mmaybe goats dont exist

yeah my issue was here:

Plain Text

        query_engine = RetrieverQueryEngine(
            retriever=retriever,
            response_synthesizer=response_synthesizer,
            node_postprocessors=[get_reranker()],
        )

I couldnt pass an llm or embed to the RQE, had to create a custom response_synthesizer to get it working

LLogan M

You could have passed the LLM to the response synthesizer and embed model to the reteriever

LLogan M

Plain Text

retriever = index.as_retriever(emebd_mode=embed_model)

from llama_index.core import get_response_synthesizer

response_synthesizer = get_response_synthesizer(response_mode="compact", llm=llm)

mmaybe goats dont exist

This is what I ended up doing

mmaybe goats dont exist

I wish there was just a way to pass a generic settings into a top level object like a chat engine that would disperse down into all the sub modules

mmaybe goats dont exist

Gets kind of complicated!

LLogan M

This was the exact thing that made the service context confusing I think 😅

LLogan M

I think now its much clearer what component is using what (but maybe thats just me)

mmaybe goats dont exist

I agree I think the part where I get a little stuck is when you are initializing something in a function that initializes something else that doesn't get the information

mmaybe goats dont exist

Like when I was initializing the retriever query engine I didn't expect a response synthesizer to be created inside of it with the default settings.

Not saying there's a quick or easy or even a clear fix, just a data point for you in the future.

I always try to be as specific as possible so I like to actually create everything manually in pass it in because I think it's much easier to understand that way, so we will continue with this approach!

LLogan M

Yea, I think the issue is a query engine is a bit of a block box. Without reading the docs extensively (or the source code), most users probably don't realize its just a wrapper around a retreiver, synthesizer, and node-postprocessors

LLogan M

probably this just points to a lack of basic api docs lol

LLogan M

which we are working on fixing!

mmaybe goats dont exist

this makes a lot of sense

mmaybe goats dont exist

i appreciate the insight

Add a reply

Find answers from the community

having an issue migrating to 0.10, for