Find answers from the community

Updated 5 months ago

Why is `LLMTextCompletionProgram` +

At a glance

The community member is experiencing slowness when using the LLMTextCompletionProgram with the GroqCloud API, despite the GroqCloud native API being fast. The community members discuss potential reasons for the slowness, with one suggesting that the program may not have a loop internally. Another community member provides step-by-step instructions on how to run the program's underlying steps, but there is no explicitly marked answer to the original question.

ppikachu8887867

Why is LLMTextCompletionProgram + GroqCloud is hella slow?? I’ve tested GroqCloud native api from terminal and it is indeed super fast, but when I use with pydantic program it is slow as hell. Does it have some kind of loop internally?

3 comments

ppikachu8887867

Plain Text

llama_llm = Groq(model="llama3-70b-8192", api_key=GROQ_API_KEY)

program = LLMTextCompletionProgram.from_defaults(
            output_cls=Data,
            prompt_template_str=prompt,
            verbose=False,
            llm=llama_llm
        )

response = program(text=source_reference_comparison, num_sections=len(sections), sections=str(sections))

LLogan M

It does not loop

LLogan M

If you wanted, you could run what its doing step-by-step too

Peeking under the hood, you could run these steps

Plain Text

program = LLMTextCompletionProgram.from_defaults(
    output_cls=Data,
    prompt_template_str=prompt,
    verbose=False,
    llm=llama_llm
)

messages = program._prompt.format_messages(
  llm=program._llm, 
  text=source_reference_comparison, 
  num_sections=len(sections), 
  sections=str(sections)
)

response = program._llm.chat(messages)

output = program._output_parser.parse(raw_output)

Add a reply