Find answers from the community

Updated 3 months ago

Hi @Logan M I am currently working on a

Hi @Logan M I am currently working on a project involving document processing and integration with an LLM. I am unsure about the correct approach for indexing with VectorStoreIndex. Specifically, I am questioning whether the way I am passing the text_splitter transformation is correct. Are there any improvements or best practices you would recommend?

Plain Text
text_splitter = SentenceSplitter(chunk_size=700, chunk_overlap=10)

# Set up LLM
Settings.llm = OpenAI(model="gpt-3.5-turbo")

index = VectorStoreIndex.from_documents(
    documents, transformation=[text_splitter]
)
W
A
2 comments
Passing transformations in a method or defining it globally both are correct and will work in all the cases.

https://docs.llamaindex.ai/en/stable/module_guides/loading/ingestion_pipeline/transformations/?h=transforma#combining-with-an-index
@WhiteFang_Jr Thank you!!
Add a reply
Sign up and join the conversation on Discord