Find answers from the community

Updated 6 months ago

Custom node embedding

At a glance

The community member is asking if there is a way to customize the embedding of a node in the llama-index library, specifically by using only certain keywords ("how", "use", "llama-index") instead of the full node text. The comments suggest that while there may not be a direct option for customizing node text embeddings, the community members discuss using techniques like tokenization, stopword filtering, and query transformations to potentially improve the vector search results. They mention specific examples and notebooks from the llama-index project that may be helpful in exploring these approaches.

Useful resources
Is there a way to custom the embedding of a node? for example the node text is "how to use llama-index?" ,I want to just use "how,use,llama-index" to calculate the embedding?
L
s
11 comments
Do you mean the query, or the actual text inside the index?
I mean the actual text inside the index and you remind me query need this too.
I am thinking if I use tokenizer and stopword filter to preprocess the text before embedding, could the vector search result be better?
Hmm It could be! Although I think Ada was probably trained on full sentences, so I'm not sure how much it will help 🤔

there is a query_transform option, to transform the query text before embedding.

I don't think there's an option that I know of for node text
Actually after reading the notebook, it seems a little weird haha but maybe it will help
haha I have read this one before.
Maybe that will help? It seems to use your idea of stopwords under the hood
wow, Thanks. I will look into it:)
haha realize that I actual read this one before when I hadn't have the idea to use stop word etc to get the better result.
Add a reply
Sign up and join the conversation on Discord