We re building an app to transform

HHK-RantTest-HarisRashid

We're building an app to transform initial data, perform redaction on what kind of data can and cannot be exposed, which ends up in an index. This data can be from various sources. In the most dummy case, imagine two sources: SQL database (Index) + List of Webpages (Index).

I'm trying to put this into a graph structure so I can query heterogeneous data sources. I'm not sure what's the problem here.

Plain Text

File ~/miniconda3/lib/python3.10/site-packages/llama_index/indices/query/base.py:34, in BaseQueryEngine.retrieve(self, query_bundle)
     33 def retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
---> 34     raise NotImplementedError(
     35         "This query engine does not support retrieve, use query directly"
     36     )

NotImplementedError: This query engine does not support retrieve, use query directly

It's asking me to use query directly, which I can't without first building a query engine.

Attachment

Screenshot_2023-06-10_at_11.50.50_AM.png

41 comments

LLogan M

Yea the SQL index doesn't have a retriever. But the graph depends on having a retriever so it's complaining. So a graph is a no-go for this.

You could also use a router query engine instead of a graph too
https://gpt-index.readthedocs.io/en/latest/examples/query_engine/RouterQueryEngine.html#router-query-engine

Currently you can mix a SQL index with a vector index:
https://gpt-index.readthedocs.io/en/latest/examples/query_engine/SQLAutoVectorQueryEngine.html
https://www.youtube.com/watch?v=ZIvcVJGtCrY

But there's also a PR to make this more flexible too, should be merged soon hopefully 🙏
https://github.com/jerryjliu/llama_index/pull/6265

HHK-RantTest-HarisRashid

This also leads me to this:
It's really confusing on what a generic solution would be where different data sources will be auto assembled into a graph structure for query. Jerry explains quite nicely towards the middle of the following video:
https://www.youtube.com/watch?v=ZIvcVJGtCrY&t=688s

But answering complex question over heterogeneous sources, can ComposableGraph do the same as SQL Auto Vector Query Engine?

ComposableGraph: https://gpt-index.readthedocs.io/en/latest/use_cases/queries.html#synthesis-over-heterogeneous-data
SQL Auto Vector Query Engine: https://gpt-index.readthedocs.io/en/latest/examples/query_engine/SQLAutoVectorQueryEngine.html

---

You're saying I should use a RouterQueryEngine, but I'm not sure if it'll be able to answer complex question that requires info from other sources.

LLogan M

Yea the SQL Auto Vector Query engine will use both sources (SQL and/or a vector index). With the incoming PR, the vector index will be able to be swapped out for anything

Currently the router query engine will only query one index

HHK-RantTest-HarisRashid

Any idea when u guys are building this PR out? That'll be really good to have as I think I badly need this to make it work over multiple sources.

LLogan M

I would expect that PR to merge over the weekend 👍

HHK-RantTest-HarisRashid

There seems to be another problem here. I can't use Azure's gpt-3.5-turbo with SQLAutoVectorQueryEngine.

An issue has also been opened for this as well.
https://github.com/jerryjliu/llama_index/issues/6266

Attachment

LLogan M

Are you using AzureChatOpenAI? Or AzureOpenAI?

LLogan M

If you are using gpt-3.5 or gpt-4, use AzureChatOpenAI

HHK-RantTest-HarisRashid

AzureChatOpenAI using gpt-3.5

HHK-RantTest-HarisRashid

Doing exactly that but its still causing problems.

Attachment

HHK-RantTest-HarisRashid

@Logan M this seems to be the only problem I'm experiencing. Thanks a lot for posting the change. I've updated my codebase with SQLJoinQueryEngine.

HHK-RantTest-HarisRashid

This solution doesn't work. Not sure why replacing a dummy engine name would work. A much neater solution would be appreciated.

Attachment

LLogan M

Yea, most of the problem is just how azure is implemented in langchain, it's not exactly user friendly to setup.

We are working on a more native solution. For now I guess it's just a debug party 😅😢

HHK-RantTest-HarisRashid

Man its a debug party, but please help me through this one.

HHK-RantTest-HarisRashid

lol

HHK-RantTest-HarisRashid

Not sure why its not passing in the prompt to AzureOpenAI endpoint. Can I build this out with custom LLM and use Azure for that? Maybe write a wrapper for it.

LLogan M

Lol I wish I could help more, but I don't have azure access just yet to test

My best advice is checking the langchain docs? Does this look any different?

llm
https://python.langchain.com/en/latest/modules/models/chat/integrations/azure_chat_openai.html?highlight=AzureChatOpenAI#azure

embeddings
https://python.langchain.com/en/latest/reference/modules/embeddings.html?highlight=Azure#langchain.embeddings.OpenAIEmbeddings

LLogan M

The LLM looks pretty different actually in their docs. Maybe that's the issue?

Something like

Plain Text

llm_predictor = LLMPredictor(llm=AzureChatOpenAI(
    openai_api_base=BASE_URL,
    openai_api_version="2023-03-15-preview",
    deployment_name=DEPLOYMENT_NAME,
    openai_api_key=API_KEY,
    openai_api_type = "azure",
)

HHK-RantTest-HarisRashid

The problem doesn't seem to be in the langchain. I can't trace the source of issue because it works fine when I directly create a query engine on an index. The same service_context works perfectly and returns response.

Attachment

Screenshot_2023-06-11_at_12.40.48_PM.png

HHK-RantTest-HarisRashid

Plain Text

from langchain.chat_models import AzureChatOpenAI


model = AzureChatOpenAI(
    openai_api_base=API_BASE,
    openai_api_version="2023-03-15-preview",
    deployment_name="Sample",
    openai_api_key=API_KEY,
    openai_api_type = "azure",
)

# LLM for Service Context
chat_llm = ChatGPTLLMPredictor(llm=model)

# Embedding Model for Service Context
embedding_llm = LangchainEmbedding(
    OpenAIEmbeddings(
        model="text-embedding-ada-002",
        deployment=EMBED_DEPLOYMENT,
        openai_api_key=API_KEY,
        openai_api_base=API_BASE,
        openai_api_type = "azure",
        openai_api_version="2023-03-15-preview",
    ),
    embed_batch_size=1,
)

service_context = ServiceContext.from_defaults(llm_predictor=chat_llm, embed_model=embedding_llm, chunk_size=1024)

HHK-RantTest-HarisRashid

Attachment

Screenshot_2023-06-11_at_12.44.49_PM.png

LLogan M

Does this thread help? Another guy found an order of operations issue (you should define embeddings first?)

LLogan M

https://discord.com/channels/1059199217496772688/1109441127918096424/1109597197898616923

HHK-RantTest-HarisRashid

Nah. Changing the order of operations doesn't help.

HHK-RantTest-HarisRashid

I'm trying to log and see the trace, it seems like LLMPredictor class is not receiving the LLM passed using ChatGPTLLMPredictor

HHK-RantTest-HarisRashid

Since its None, it by default picks the OpenAI one and hence throws exception.

LLogan M

Oh tbh I missed that you were using that class, I'm pretty sure it's deprecated (or supposed to be anyways lol)

Does it work if you use LLMPredictor instead?

HHK-RantTest-HarisRashid

Attachment

HHK-RantTest-HarisRashid

It works fine if I directly augment it in the code with AzureChatOpenAI

HHK-RantTest-HarisRashid

Lemme check

HHK-RantTest-HarisRashid

It picks it up when I first define it. Ofc, I'm passing it in, it will. It still throws the same error.

Attachment

HHK-RantTest-HarisRashid

This is what ends up happening.

Attachment

HHK-RantTest-HarisRashid

I think you'll need my notebook to diagnose this, not sure if you're understanding it right.

LLogan M

I mean, I can see whats happening lol it's defaulting back to openai for ... unknown reasons lol

You know, we recently introduced a global service context which might actually help

Rather than passing the service context into everything, you can try

Plain Text

from llama_index import set_global_service_context

....

set_global_service_context(service_context)

This is a sort of catchall, because anytime the code does ServiceContext.from_defaults(), it will use that service context instead

HHK-RantTest-HarisRashid

You got a guide on this. Let me quickly swap out and see if it works.

HHK-RantTest-HarisRashid

Got it!!

HHK-RantTest-HarisRashid

Works.

HHK-RantTest-HarisRashid

I think for now, it serves the purpose, but later down the road, in case I'll need to use different service_contexts for different indexes, etc, we'll have to see so it doesn't throw an error again.

LLogan M

Yea exactly. Something somewhere is causing it to reset back to defaults.

I'll be able to look into this more this week 💪

LLogan M

Glad it works for now though!

HHK-RantTest-HarisRashid

U guys are awesome. Fast help. Thanks for all the work u guys are doing 💪

Add a reply

Find answers from the community

We re building an app to transform