turns out claude responses can be empty

At a glance

The community member encountered an issue where Claude responses could be empty when sending completed message sequences. They provided a temporary "monkey patch" fix to the AnthropicProvider.get_text_from_response function in the llama_index library. The issue seems to be specific to the "bedrock space" and occurs when connecting directly to the Anthropic API. The community member was testing an edge case involving generating text larger than 4,000 tokens by making recursive calls with a max token limit of 5. They plan to share a cleaned-up notebook to demonstrate the issue further.

ssansmoraxz

turns out claude responses can be empty if you are sending completed messages sequence. I monkey patched an interceptor there.
otherwise:-

Plain Text

File ~/.local/lib/python3.10/site-packages/llama_index/llms/bedrock/utils.py:150, in AnthropicProvider.get_text_from_response(self, response)
    149 def get_text_from_response(self, response: dict) -> str:
--> 150     return response["content"][0]["text"]

IndexError: list index out of range

should probably have some secondstage validation.

temporary money patch fixed:

Plain Text

import llama_index.llms.bedrock.utils
def f(self, response):
    if len(response["content"]) > 0:
        return response["content"][0]["text"]
    else:
        return ''
llama_index.llms.bedrock.utils.AnthropicProvider.get_text_from_response = f

Attachment

10 comments

RRoland Tannous

what's the context ? i used claude with llama-index agents and retrieval pipelines. never had that issue

WWhiteFang_Jr

Seems like this is happening from bedrock space

RRoland Tannous

i am connecting directly to anthropic's api

RRoland Tannous

so that must be why

ssansmoraxz

tbh it was very much an edge case I was testing something else on. I was taking max_token as 5.
and making recursive calls to generate a complete output by appending the previous generations.

ssansmoraxz

lemme clean up the notebook and share it with you guys.

ssansmoraxz

not sure about the anthropic or gcp side, but it is an edgecase in bedrock at least.

RRoland Tannous

ah interesting

RRoland Tannous

what is the objective (I know it's besides the point)

ssansmoraxz

generating texts larger than 4k tokens

Add a reply

Find answers from the community

turns out claude responses can be empty