Find answers from the community

Updated 3 months ago

having an issue migrating to 0.10, for

having an issue migrating to 0.10, for some reason, even though im passing the embeding model to my query engines and retrievers, its still complaining i dont have an open_ai key set, as its trying to use that as a default

This is ok because im not using openAI, but is there a better way to set this up? I saw something about settings but not sure. Quite annoying!
L
r
m
21 comments
Set the global default, this is the way πŸ™
Plain Text
from llama_index.core import Settings

Settings.llm = ...
Settings.embed_model = ...
does this replace set_global_service_context , a method I cannot find anymore?
The issue is that these change per chat, so if a user wants to chat with a different model I don't really want to update global settings
That method is still there (from llama_index.core import set_global_service_context). And under the hood that method does the above
Then you need to rely on passing into the interfaces

For example

Plain Text
index = VectorStoreIndex.from_documents(..., embed_model=embed_model)

index.as_chat_engine(..., llm=llm)
yeah my issue was here:
Plain Text
        query_engine = RetrieverQueryEngine(
            retriever=retriever,
            response_synthesizer=response_synthesizer,
            node_postprocessors=[get_reranker()],
        )

I couldnt pass an llm or embed to the RQE, had to create a custom response_synthesizer to get it working
You could have passed the LLM to the response synthesizer and embed model to the reteriever
Plain Text
retriever = index.as_retriever(emebd_mode=embed_model)

from llama_index.core import get_response_synthesizer

response_synthesizer = get_response_synthesizer(response_mode="compact", llm=llm)
This is what I ended up doing
I wish there was just a way to pass a generic settings into a top level object like a chat engine that would disperse down into all the sub modules
Gets kind of complicated!
This was the exact thing that made the service context confusing I think πŸ˜…
I think now its much clearer what component is using what (but maybe thats just me)
I agree I think the part where I get a little stuck is when you are initializing something in a function that initializes something else that doesn't get the information
Like when I was initializing the retriever query engine I didn't expect a response synthesizer to be created inside of it with the default settings.

Not saying there's a quick or easy or even a clear fix, just a data point for you in the future.

I always try to be as specific as possible so I like to actually create everything manually in pass it in because I think it's much easier to understand that way, so we will continue with this approach!
Yea, I think the issue is a query engine is a bit of a block box. Without reading the docs extensively (or the source code), most users probably don't realize its just a wrapper around a retreiver, synthesizer, and node-postprocessors
probably this just points to a lack of basic api docs lol
which we are working on fixing!
this makes a lot of sense
i appreciate the insight
Add a reply
Sign up and join the conversation on Discord