anthropic-cookbook/third_party/LlamaInde...

mmingsheng

Hello,
I am trying out the code snippet obtained from claude https://github.com/anthropics/anthropic-cookbook/blob/main/third_party/LlamaIndex/Basic_RAG_With_LlamaIndex.ipynb

Instead of using claude, I am trying to use bedrock services claude-sonnet model and cohere-english-v3. But the problem is that it will fail whenever I am try to create vectorstore index with the following error:
ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #/texts/1: expected maxLength: 2048, actual: 2166#/texts/2: expected maxLength: 2048, actual: 2312#/texts/3: expected maxLength: 2048, actual: 2315#/texts/4: expected maxLength: 2048, actual: 2151#/texts/6: expected maxLength: 2048, actual: 2072#/texts/7: expected maxLength: 2048, actual: 2164#/texts/8: expected maxLength: 2048, actual: 2091#/texts/9: expected maxLength: 2048, actual: 2201, please reformat your input and try again.

My code:
from llama_index.llms.bedrock import Bedrock
from llama_index.embeddings.bedrock import BedrockEmbedding
from llama_index.core import Settings
from llama_index.core import (
VectorStoreIndex,
SimpleDirectoryReader,
)
llm = Bedrock(model="anthropic.claude-3-sonnet-20240229-v1:0", region_name='us-west-2')
embed_model = BedrockEmbedding(service_name="bedrock-runtime", model="cohere.embed-english-v3", region_name='us-west-2')

Settings.llm = llm
Settings.embed_model = embed_model
Settings.chunk_size = 512

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents,)
query_engine = index.as_query_engine(similarity_top_k=3)
response = query_engine.query("What did author do growing up?")
Am I doing something wrong ?

11 comments

WWhiteFang_Jr

Your node size is not accurate according to me. But code wise it looks okay to me. It should work!

For the sake of it. lets try creating nodes by ourselve before putting it into for creating index.

Plain Text

from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter

node_parser = SentenceSplitter(chunk_size=1024, chunk_overlap=20)

nodes = node_parser.get_nodes_from_documents(
    [Document(text="long text")], show_progress=False
)

# Pass the nodes for creating ondex
index= VectorStoreIndex(nodes)
# then proceed further

mmingsheng

Hey, appreciate your help
If I just use your code as is, it will work as expected
It will fail whenever I try to load with simple directory reader

I checked the documents returned is valid but it causes the index to fail for some reason.

Do you have any idea why this happened?

Attachment

WWhiteFang_Jr

Strange just tried with simpledirectory and it worked for me!

Could you try this once:

Plain Text

from llama_index.llms.bedrock import Bedrock
from llama_index.embeddings.bedrock import BedrockEmbedding
from llama_index.core import Settings
from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
)
llm = Bedrock(model="anthropic.claude-3-sonnet-20240229-v1:0", region_name='us-west-2')
embed_model = BedrockEmbedding(service_name="bedrock-runtime", model="cohere.embed-english-v3", region_name='us-west-2')

Settings.llm = llm
Settings.embed_model = embed_model
Settings.chunk_size = 512

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

# iterate over the created nodes
for node in index.docstore.docs:
  print(node) # check on the text size here

mmingsheng

it will just fail with the same error

Attachment

mmingsheng

llama-index Version: 0.10.29
llama-index-embeddings-bedrock Version: 0.1.4

mmingsheng

i tried to update all my libraries and its still the same

WWhiteFang_Jr

Hmm wierd, can you share the whole traceback once

mmingsheng

yeah sure, let me know if you need anything else.

LLogan M

cohere has a tiny size limit for llms, and the token counting isn't perfect. Manually lower it a bit

llm = Bedrock(..., context_size=2000, max_tokens=256)

LLogan M

try that

mmingsheng

Hello Logan thanks for tips I had to set context size to 1800 to make it runnable
but now the program will not stop running
Maybe I will just use other embedding models instead

Add a reply

Find answers from the community

anthropic-cookbook/third_party/LlamaInde...