Find answers from the community

Updated 5 months ago

Hi everyone , I am benchmarking time it

Hi everyone , I am benchmarking time it takes to chunk , embed documents and then insert into vector database .

Following code does everything in 1 shot. How can I split into different components ? . Please guide.

index = VectorStoreIndex.from_documents(
documents,
storage_context=storage_context,
)
L
P
2 comments
Plain Text
from llama_index.embedding.openai import OpenAIEmbedding
from llama_index.core.node_parser import SentenceSplitter

nodes = SentenceSplitter()(documents)
embeddings = 

embed_model.get_text_embedding_batch([node.get_content(metadata_mode="embed") for node in nodes])
for (node, embedding) in zip(nodes, embeddings):
  node.embedding = embedding

storage_context.vector_store.add(nodes)

# if you are using the default vector store, you might also want to add to the docstore
storage_context.docstore.add_documents(nodes)

index = VectorStoreIndex(nodes=[], storage_context=storage_context)
Add a reply
Sign up and join the conversation on Discord