Find answers from the community

Updated 3 months ago

Responses cut off

Hello, I'm having a problem that gpt responses are cut off in python, how can I fix it?

11 comments

Just added to the brand new faq today

https://docs.google.com/document/d/1bLP7301n4w9_GsukIYvEhZXVAvOMWnrxMy089TYisXU/edit?usp=sharing

AAlanovski

thanks Logan

AAlanovski

but still cutting, i'm using GPTSimpleVectorIndex

AAlanovski

I used llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-davinci-002", max_tokens=1024))

LLogan M

Do you pass the predictor in when you both load from disk or construct the index?

AAlanovski

llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-davinci-003", max_tokens=4096))

# define prompt helper
# set maximum input size
max_input_size = 4096
# set number of output tokens
num_output = 2048
# set maximum chunk overlap
max_chunk_overlap = 20
prompt_helper = PromptHelper(max_input_size, num_output, max_chunk_overlap)

index = GPTSimpleVectorIndex(
    documents, llm_predictor=llm_predictor, prompt_helper=prompt_helper
)

LLogan M

Ah there's the problem!

You've set max_tokens to 4096 in the predictor, which isn't actually possible

Meanwhile, num_output is set to 2048

Try setting both to 2048

AAlanovski

# Save your index to a index.json file
index.save_to_disk('index.json')
# Load the index from your saved index.json file
index = GPTSimpleVectorIndex.load_from_disk('index.json')

LLogan M

And when you load from disk, pass the prompt helper and llm predictor back in

AAlanovski

how?

LLogan M

load_from_disk("index.json", llm_predictor=llm_predictor, prompt_helper=prompt_helper)

Add a reply