Langchain

AAbhishek

Hey Team, Hope you're doing all good 🙂

We recorded an issue while querying the index, We haven't faced it earlier and isn't reproducible from our side. We need assistance on debugging it. Sharing below the necessary details:

Version details:
llama-index==0.8.38
langchain==0.0.304

Traceback message in the attached file
@Logan M / @ravitheja any help here?

6 comments

LLogan M

Seems like the Openai API failed to start streaming?

It looks like you are using a langchain LLM by the way, I would recommend using a llama index LLM

Plain Text

from llama_index.llms import OpenAI

llm = OpenAI(model="gpt-3.5-turbo", temperature=0.1)

AAbhishek

Okay, Got it. But it won't be easy task for us because we use multiple LLMs. 2 of them are azure openai which are hosted on different zones and then we have a LLM which is connected with openai and then they all uses portkey proxy PLUS we also have embedding models from azure openai hosted on different zones.

Thanks, Will see if we can move towards using llama-index llms.

LLogan M

No worries if it's a huge task -- I think the original error is just a random openai API issue 🤔

AAbhishek

Yep, It looks random and it's very rare because we have encountered it only once. Will still check if we could move our integrations.

AAbhishek

Hi Logan, I was able to work out with llama-index LLM except I'm not able to understand one last thing.

Here's the pervious code for LLM:

Plain Text

from langchain.chat_models import ChatOpenAI

self.llm = ChatOpenAI(
    model_name=self.model_name,
    temperature=self.temperature,
    model_kwargs=model_kwargs,
    max_tokens=self.max_output_tokens,
    streaming=True,
    openai_api_key=openai_api_key,
)
predictor = LLMPredictor(llm=self.llm)

Updated it with Llama-index llm:

Plain Text

from llama_index.llms import AzureOpenAI
self.llm = AzureOpenAI(
    model=self.model,
    temperature=self.temperature,
    max_tokens=self.max_output_tokens,
    enginer=self.azure_chat_engine_name,
    additional_kwargs=model_kwargs,
    api_key=azure_openai_key,
)
self.llm_predictor = LLMPredictor(llm=self.llm)

However, I'm not able to figure out How to setup streaming=True. Do i need to call stream chat method, If so how would it work when instead of calling it with llm_predictor. We use custom_index.query()

LLogan M

No need to set streaming=True in the LLM

You should just be able to set streaming=True in your query engine.

Or, if you are using an agent or chat engine, you can use stream chat 👍

Add a reply

Find answers from the community

Langchain