Find answers from the community

Updated 8 months ago

anthropic-cookbook/third_party/LlamaInde...

Hello,
I am trying out the code snippet obtained from claude https://github.com/anthropics/anthropic-cookbook/blob/main/third_party/LlamaIndex/Basic_RAG_With_LlamaIndex.ipynb

Instead of using claude, I am trying to use bedrock services claude-sonnet model and cohere-english-v3. But the problem is that it will fail whenever I am try to create vectorstore index with the following error:
ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #/texts/1: expected maxLength: 2048, actual: 2166#/texts/2: expected maxLength: 2048, actual: 2312#/texts/3: expected maxLength: 2048, actual: 2315#/texts/4: expected maxLength: 2048, actual: 2151#/texts/6: expected maxLength: 2048, actual: 2072#/texts/7: expected maxLength: 2048, actual: 2164#/texts/8: expected maxLength: 2048, actual: 2091#/texts/9: expected maxLength: 2048, actual: 2201, please reformat your input and try again.

My code:
from llama_index.llms.bedrock import Bedrock
from llama_index.embeddings.bedrock import BedrockEmbedding
from llama_index.core import Settings
from llama_index.core import (
VectorStoreIndex,
SimpleDirectoryReader,
)
llm = Bedrock(model="anthropic.claude-3-sonnet-20240229-v1:0", region_name='us-west-2')
embed_model = BedrockEmbedding(service_name="bedrock-runtime", model="cohere.embed-english-v3", region_name='us-west-2')

Settings.llm = llm
Settings.embed_model = embed_model
Settings.chunk_size = 512

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents,)
query_engine = index.as_query_engine(similarity_top_k=3)
response = query_engine.query("What did author do growing up?")
Am I doing something wrong ?
W
m
L
11 comments
Your node size is not accurate according to me. But code wise it looks okay to me. It should work!

For the sake of it. lets try creating nodes by ourselve before putting it into for creating index.

Plain Text
from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter

node_parser = SentenceSplitter(chunk_size=1024, chunk_overlap=20)

nodes = node_parser.get_nodes_from_documents(
    [Document(text="long text")], show_progress=False
)

# Pass the nodes for creating ondex
index= VectorStoreIndex(nodes)
# then proceed further
Hey, appreciate your help
If I just use your code as is, it will work as expected
It will fail whenever I try to load with simple directory reader

I checked the documents returned is valid but it causes the index to fail for some reason.

Do you have any idea why this happened?
Attachment
image0.jpg
Strange just tried with simpledirectory and it worked for me!

Could you try this once:
Plain Text
from llama_index.llms.bedrock import Bedrock
from llama_index.embeddings.bedrock import BedrockEmbedding
from llama_index.core import Settings
from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
)
llm = Bedrock(model="anthropic.claude-3-sonnet-20240229-v1:0", region_name='us-west-2')
embed_model = BedrockEmbedding(service_name="bedrock-runtime", model="cohere.embed-english-v3", region_name='us-west-2')

Settings.llm = llm
Settings.embed_model = embed_model
Settings.chunk_size = 512

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

# iterate over the created nodes
for node in index.docstore.docs:
  print(node) # check on the text size here
it will just fail with the same error
Attachment
llama_index.png
llama-index Version: 0.10.29
llama-index-embeddings-bedrock Version: 0.1.4
i tried to update all my libraries and its still the same
Hmm wierd, can you share the whole traceback once
yeah sure, let me know if you need anything else.
cohere has a tiny size limit for llms, and the token counting isn't perfect. Manually lower it a bit

llm = Bedrock(..., context_size=2000, max_tokens=256)
Hello Logan thanks for tips I had to set context size to 1800 to make it runnable
but now the program will not stop running
Maybe I will just use other embedding models instead
Add a reply
Sign up and join the conversation on Discord