Batch size

Getting a "batch size should not be larger than 2048" error, but only when use_async=True

Offending Line:

Plain Text

VectorStoreIndex(nodes, storage_context=storage_context, use_async=True)

Error:

Plain Text

File ~/superwallet/env/lib/python3.11/site-packages/tenacity/_asyncio.py:50, in AsyncRetrying.__call__(self, fn, *args, **kwargs)
     49 try:
---> 50     result = await fn(*args, **kwargs)
     51 except BaseException:  # noqa: B902

File ~/superwallet/env/lib/python3.11/site-packages/llama_index/embeddings/openai.py:167, in aget_embeddings(list_of_text, engine, **kwargs)
    158 """Asynchronously get embeddings.
    159 
    160 NOTE: Copied from OpenAI's embedding utils:
   (...)
    165 
    166 """
--> 167 assert len(list_of_text) <= 2048, "The batch size should not be larger than 2048."
    169 # replace newlines, which can negatively affect performance.

AssertionError: The batch size should not be larger than 2048.

10 comments

LLogan M

Oh that's super weird. All our embedding code is basically copied from openai lol

I'll have to try and reproduce this to track it down at some point today.

TTurtles

Thanks @Logan M

jjjmachan

can you run len(nodes) to get the number of nodes you have?

TTurtles

@jjmachan 11,241. Would you advise manually batching these?

TTurtles

@jjmachan Is that a lot?

TTurtles

And @Logan M any luck?

LLogan M

I haven't had a chance yet 😅 I'm assuming you are using async because there's so many nodes right?

You can try increasing the batch size and avoid using async

Plain Text

from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index import ServiceContext

service_context = ServiceContext.from_defaults(embed_model=OpenAIEmbedding(embed_batch_size=2000))

TTurtles

For the WeaviateVectorStore (link below), what do the class_prefix, class_name, index_name, and text_key do? The documentation is a bit sparse

https://gpt-index.readthedocs.io/en/latest/reference/storage/vector_store.html#llama_index.vector_stores.WeaviateVectorStore

TTurtles

This helped a lot btw

LLogan M

Yea they aren't documented well lol

Looking at the code, I wouldn't change text key

The rest seem to be there to help organize how the data is stored by augmenting the index name

You can see the source code here
https://github.com/jerryjliu/llama_index/blob/d9bdd8c55f29837d41b037df41799a5765cdd20d/llama_index/vector_stores/weaviate.py#L50

Add a reply

Find answers from the community

Batch size