what could the the reason the metadata

Are you using a vector db integration ?

using LanceDBVectorStore

creating index like this. VectorStoreIndex.from_documents(all_docs, storage_context=storage_context)

i think when i did VectorStoreIndex(all_docs) it retained the original urls in the metadata. using SitemapReader

i see raise ValueError("Metadata filters not implemented for LanceDB yet.")

oops!

line 107 of lancedbvectorstore

do you recommend anything else which has metadata and easily deployable in a container+good search capabilities

I am looking to get original html page links to refer in the response (as source of truth)

if dont use any vector store, it was painfully slow for tree summarization!

this is giving faster results but I am losing original source

ah lancedb isn't used as often. I see that it wasn't updated to re-construct the metadata when querying :PepeHands: I can hopefully fix that tomorrow

even weaviate doesn't have it

thank you!

which vector store (Free ones) is good?!

Weaviate definitely should have it 🤔🤔 at least in the latest versions of the code.

Qdrant is another good popular one

i didnt test it. i was looking at source code and saw similar exception on metadata not implemented.

but plz let me know when you have lancedb implemented with metadata. seems to be working excellent for my use case

@Logan M plz let me know when you have lancedb metadata working

Hey, figured I'd bring this back up instead of making a new thread. @Logan M I was playing with LanceDB and was trying to get filtering to work: https://github.com/run-llama/llama_index/blob/5e01f9c60c1516df35170fa45f36a848dd82353e/llama-index-core/llama_index/core/vector_stores/types.py#L86

Even using "pure lancedb" to try and filter on a vectorstore created by LlamaIndex fails as well: https://lancedb.github.io/lancedb/sql/#filtering-without-vector-search

But it seems the way the vector store is constructed is it dumps all the metadata into a new column, and then lancedb can't query it as it excepts a flat column (not a json value). I followed (and updated to use the latest llamaindex lib) this example: https://docs.llamaindex.ai/en/stable/examples/vector_stores/postgres/#apply-metadata-filters which I bet works fine for postgres still.

On a related topic, would it be "easy" to add native hybrid integration, like pinecone (vector_store_query_mode="hybrid")? LanceDB support FTS with Tantivy...not sure if there's any performance benefits to doing it on the llamaindex side, or just construct it with a fusion retriever: https://lancedb.github.io/lancedb/fts/

Update to the thread. Got it to work. Just needed to add metadata. to the key field. So if the field name is filename, in postgres you'd just call that, with Lance you call metadata.filename. @explorer