Find answers from the community

Updated last month

Troubleshooting Issues With Llm.achat

At a glance

A community member is experiencing issues with the llm.achat function. Their chat messages include a system prompt and a user request to create a summary of a 18000 token document that is at least 2500 tokens long. However, the function creates the response and uses the tokens, but then hangs indefinitely. Another community member comments that they were testing on the latest versions of things.

Hi there again,

having issues with the llm.achat at the moment.

My chat messages look like

[ChatMessage(role=<MessageRole.SYSTEM: 'system'>, content='[system_prompt]', additional_kwargs={}),
ChatMessage(role=<MessageRole.USER: 'user'>, content=[
{'text': '<document>{18000 token document}</document>', 'type': 'text', 'cache_control': {'type': 'ephemeral'}},
{'text': 'Please create a summary of the document that is at least 2500 tokens long. Currently the document is 18000 tokens long.', 'type': 'text'}
], additional_kwargs={})]

Executed by:
llm.achat(
messages,
extra_headers={
"anthropic-beta": "prompt-caching-2024-07-31",
},
),

It creates the response and uses the tokens, but hangs indefinitely. I can't copy the exact text due to privacy concerns
Attachment
image.png
L
1 comment
(I was testing on the latest versions of things though)
Add a reply
Sign up and join the conversation on Discord