Find answers from the community

Updated 3 months ago

Hi GuysOn PgVectorStore with Hybrid Search, we try to create sql index for the tsv colum

Hi Guys

On PgVectorStore with Hybrid Search, we try to create sql index for the tsv column, that's why we got this bug.
https://github.com/jerryjliu/llama_index/issues/7740

Any idea why we don't built SQL index for the main vector embedding column? FYI PgVector itself support indexing with IVFFlat & HNSW
https://github.com/pgvector/pgvector#indexing

I tried it myself, the performance improvement is great, but the hardware utilization also increase significantly, especially when inserting new document. But probably I did that wrongly

WDYT guys ?
k
L
R
5 comments
Please stand by @Rendy Febry as I search the knowledge base. I normally take about 20 seconds or so. :wave:
----------------------
I'm sorry, but the information extracts provided do not contain specific details about why SQL index is not built for the main vector embedding column in PgVectorStore with Hybrid Search. It's also important to note that the specific issue you mentioned () is not referenced in the extracts.

However, it's worth noting that indexing can indeed improve query performance but at the cost of increased storage space and potentially slower write operations, as you've experienced. The decision to index or not may depend on various factors including the specific use case, the volume of data, the frequency of write operations versus read operations, and hardware constraints.

For more detailed information or guidance, I would recommend referring to the official documentation or reaching out to the maintainers of the project directly.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------
@Rendy Febry These are new features to pgvector -- so we havent added this yet. Could definitely be an optional param to enable!
Oh I'm sorry for tagging you directly, I forgot this is not my company slack channel. πŸ™
Oh really, I though they already have index from beginning, at least the IVFFlat one.

But yeah, looks like they still have a few issues on their indexing system
https://github.com/pgvector/pgvector/blob/master/CHANGELOG.md
Add a reply
Sign up and join the conversation on Discord