Find answers from the community

Updated 2 months ago

@kapa.ai Is it possible to specify a

Is it possible to specify a node parser at the time of insertion into an index? If not, what is the recommended way of handling the insertion of multiple different document types that require different node parsers?

I have several different node parsers that are optimized for chunking different types of documents and I need to be able to hot-swap them out depending on which type of document I am inserting without rebuilding the index from scratch every time. I am assuming that creating an index is not cheap. What is the recommended way of doing this?
L
a
4 comments
Parse the nodes before inserting imo
Plain Text
node_parser = ...
nodes = node_parser(documents)

index.insert_nodes(nodes)
Then i would remove the node_parser from the transformations array correct?

transformations=[node_parser, embedding_component.embedding_model],
You could keep it if you wanted, but it wouldn't be used
Add a reply
Sign up and join the conversation on Discord