LlamaIndex
Log in

Find answers from the community

Updated 11 months ago

indexing a fat dataset

DDangFutures
·
indexing a fat dataset
are there any libareis to speed up the processing. use bge large on an A100
Attachment
image.png
L
D
4 comments
LLogan M
Are you using local embeddings? Or they are hosted on a server over an API?
DDangFutures
was using local bge
LLogan M
You can definitely increase the batch size of embeddings.

If you run this using an IngestionPipeline object, you can also enable multithreading (which might help a lot here too)
LLogan M
https://docs.llamaindex.ai/en/stable/examples/ingestion/parallel_execution_ingestion_pipeline.html#define-our-ingestionpipeline
Add a reply
Sign up and join the conversation on Discord