having an issue migrating to 0.10, for

At a glance

having an issue migrating to 0.10, for some reason, even though im passing the embeding model to my query engines and retrievers, its still complaining i dont have an open_ai key set, as its trying to use that as a default

This is ok because im not using openAI, but is there a better way to set this up? I saw something about settings but not sure. Quite annoying!

21 comments

LLogan M

Set the global default, this is the way 🙏

LLogan M

Plain Text

from llama_index.core import Settings

Settings.llm = ...
Settings.embed_model = ...

rrobwhelanatx

does this replace set_global_service_context , a method I cannot find anymore?

mmaybe goats dont exist

The issue is that these change per chat, so if a user wants to chat with a different model I don't really want to update global settings

LLogan M

That method is still there (from llama_index.core import set_global_service_context). And under the hood that method does the above

LLogan M

Then you need to rely on passing into the interfaces

For example

Plain Text

index = VectorStoreIndex.from_documents(..., embed_model=embed_model)

index.as_chat_engine(..., llm=llm)

mmaybe goats dont exist

yeah my issue was here:

Plain Text

        query_engine = RetrieverQueryEngine(
            retriever=retriever,
            response_synthesizer=response_synthesizer,
            node_postprocessors=[get_reranker()],
        )

I couldnt pass an llm or embed to the RQE, had to create a custom response_synthesizer to get it working

LLogan M

You could have passed the LLM to the response synthesizer and embed model to the reteriever

LLogan M

Plain Text

retriever = index.as_retriever(emebd_mode=embed_model)

from llama_index.core import get_response_synthesizer

response_synthesizer = get_response_synthesizer(response_mode="compact", llm=llm)

mmaybe goats dont exist

This is what I ended up doing

mmaybe goats dont exist

I wish there was just a way to pass a generic settings into a top level object like a chat engine that would disperse down into all the sub modules

mmaybe goats dont exist

Gets kind of complicated!

LLogan M

This was the exact thing that made the service context confusing I think 😅

LLogan M

I think now its much clearer what component is using what (but maybe thats just me)

mmaybe goats dont exist

I agree I think the part where I get a little stuck is when you are initializing something in a function that initializes something else that doesn't get the information

mmaybe goats dont exist

Like when I was initializing the retriever query engine I didn't expect a response synthesizer to be created inside of it with the default settings.

Not saying there's a quick or easy or even a clear fix, just a data point for you in the future.

I always try to be as specific as possible so I like to actually create everything manually in pass it in because I think it's much easier to understand that way, so we will continue with this approach!

LLogan M

Yea, I think the issue is a query engine is a bit of a block box. Without reading the docs extensively (or the source code), most users probably don't realize its just a wrapper around a retreiver, synthesizer, and node-postprocessors

LLogan M

probably this just points to a lack of basic api docs lol

LLogan M

which we are working on fixing!

mmaybe goats dont exist

this makes a lot of sense

mmaybe goats dont exist

i appreciate the insight

Add a reply

Find answers from the community

having an issue migrating to 0.10, for