Find answers from the community

Updated 3 months ago

Response length, is this a limitation of

Response length, is this a limitation of something or a setting to be adjusted?

5 comments

You can adjust it with max_tokens in service context. The LLMs do have response length limits though

Example:

Plain Text

service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-3.5-turbo", temperature=0, max_tokens=1500))

I have my max tokens at 2048. I believe it's giving a longer response but when it gets to the end of the box it cuts it off maybe.

I'm using hf models for embeddings and llm both lol

i have it running at http://212.178.231.251:8501/ if you would like to see it in action

Add a reply