Transitioning from LLM OpenAI to AzureOpenAI gpt4o deployment with token output limitations
Transitioning from LLM OpenAI to AzureOpenAI gpt4o deployment with token output limitations
At a glance
The community member transitioned from using OpenAI's LLM to Azure's OpenAI GPT-4 deployment, but they are unable to get the model to produce more than 1000 tokens. They have not set the max_tokens parameter and confirmed it is None in their settings. Other community members suggest that Azure may handle the lack of max_tokens setting differently than OpenAI, and recommend trying to set it to a value like 2000. However, this did not resolve the issue. One community member noted that the multimodal Azure LLM has a max_new_tokens setting of 300, which may be the cause of the limitation.
I transitioned from LLM OpenAI to AzureOpenAI gpt4o deployment, but I can't get the model to produce more than 1000 tokens. I have not set up max_tokens and confirmed its None in Settings.llm. Not sure what settings im missing here. any one experience the same?
Pretty sure if max tokens isn't set, it sends it as None to the api. Maybe openai handles that differently than azure? Have you tried actually setting it to a value like 2000?
