and what llama-index version do you have?
(fyi, no one else has had this issue, so pretty confused on whats causing it lol)
Shouldn't it be model in place of mode? At the LLM declaration line.
I'm sure this wont be the issue at your side because LlamaIndex sets a default model if not provided.
the setup guide says mode
Oh okay maybe the previous version could have had mode
I tried with 0.7.16 with Azure for response generation - Able to stream
Worked for 0.7.20 as well.
Here is the code part
embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2"))
llm=AzureOpenAI(engine="engine", model="gpt-35-turbo", temperature=0.0)
service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)
documents = SimpleDirectoryReader(input_files=["LlamaIndex/paul_graham_essay.txt"], filename_as_id=True ).load_data()
index = GPTVectorStoreIndex.from_documents(documents, service_context=service_context)
query_engine = index.as_query_engine(streaming=True)
streaming_response = query_engine.query(
"What did the author do growing up?",
)
print(streaming_response.print_response_stream())This is the error I am getting
I think mode might be a typo 😅 try changing it to model, or copying the code above 👍
using model didn't fix, the error is occurring in the openai source file however i am using azure openai?? does this mean the code isn't using azure openai
The azure class is just a light wrapper on openai. It just does extra env checking. They use the same API under the hood
I'm not sure what the issue is here, but since @WhiteFang_Jr got it to work, I'm going to assume it's something on your end 😅 Copying his code exactly should work no? If not, I would try starting with a fresh venv
I'm using that code and it's saying i need to include deployment_id in the embedding model but when I do that it says the deployment doesn't exist - even though it does.
It says deployment and not deployment_id, Can you check if the key is correct?
# You need to deploy your own embedding model as well as your own chat completion model
embedding_llm = LangchainEmbedding(
OpenAIEmbeddings(
model="text-embedding-ada-002",
deployment="<insert EMBEDDING model deployment name from azure>",
openai_api_key=openai.api_key,
openai_api_base=openai.api_base,
openai_api_type=openai.api_type,
openai_api_version=openai.api_version,
),
embed_batch_size=1,
)The key is correct. The code you've just sent I have already tried and it didn't work. It's from the documentation. It works without streaming but I get an error when using streaming, that is the original issue I was having.
print(streaming_response.print_response_stream())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\James\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\response\schema.py", line 79, in print_response_stream
for text in self.response_gen:
File "C:\Users\James\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\llm_predictor\utils.py", line 28, in gen
for response in chat_response_gen:
File "C:\Users\James\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\llms\base.py", line 157, in wrapped_gen
for x in f_return_val:
File "C:\Users\James\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\llms\openai.py", line 206, in gen
delta = response["choices"][0]["delta"]
~~~~~~~^^^
IndexError: list index out of range
Did you try with a fresh venv?
I checked with your exact version also, Did not face this issue with Azure setuup
i'm getting the same exact error with azure openai api, and not with standard openai...
Yes tried this, had no effect.
Can you try with HF embeddings once. In place of Azure embeddings
embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2"))
Hi, I tried this and got the same streaming error.
Can you create a fresh venv and try with the latest llamaindex version?
Problem with your issues is that it has not occurred till now to anyone and is working fine at my end also.
Tried fresh venv, no luck. very confusing
Did you try with latest llamaindex version?
Can you share your whole code with HF embeddings
Remove any API key if present in the code
Hey you have enabled streaming in
LLMPredictor class as True and then while creating query_engine you have set it to False. I don't think we need to pass streaming variable in
Azure Class.
No need to create
LLMPredictor class now, Simply pass the LLM into
service_contextAlso I'm assuming
settings["doc"] provides the folder path where your docs are present. when you laod documents like this.
documents = SimpleDirectoryReader(settings["doc"], filename_as_id=True ).load_data()I have made some changes in the code. Worked at my end with these changes.
llm = AzureOpenAI(engine = "gpt35turbo", model="gpt-35-turbo-16k")
embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2"))
service_context = ServiceContext.from_defaults(llm=llm,embed_model=embed_model)
set_global_service_context(service_context)
documents = SimpleDirectoryReader(settings["doc"], filename_as_id=True ).load_data()
index = GPTVectorStoreIndex.from_documents(documents, service_context=service_context)
query_engine = index.as_query_engine(streaming=True)
streaming_response = query_engine.query("Query based on your Dataset here!")
print(streaming_response.print_response_stream())Try with this
ah sorry having it set to false was for testing it wthout streaming, just copied this code and the issue remains. Also jus tried updating Langchain, clearly something on my end. Could it be an issue with the api_base I am using?
api_base will be given in your Azure console for OpenAI. You can check there for this value
just found a mistake and corrected it, now I get an error saying AzureOpenAI has no attribute 'stream'
Use this code, Could be because you have defined streaming in your AzureOpenAI class in your code
Ok copied it exactly but still getting this error: File "C:\Users\James\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\llms\openai.py", line 229, in gen
delta = response["choices"][0]["delta"]
~~~~~~~^^^
IndexError: list index out of range
Lol, seems like you will have to debug it more. I think using a debugger if using vs code and putting stopper in the files where you getting error can help you to gain more insights
I blame the azure deployment at this point 🥲
OK, thanks lots for your help trying to figure this out 👍
