Hi everyone, how can i get the all nodes

At a glance

A community member is trying to retrieve all nodes from an existing PG vector store to use for a BM25 retriever. They have tried using VectorStoreIndex.from_vector_store(vector_store=vector_store) and index.docstore.docs.values(), but it returns an empty dictionary even though there are values in the table.

Another community member suggests that the nodes are not present locally, but are stored in the vector store. They recommend using the PG client to fetch all the data from the vector store and then feeding it to the BM25 retriever. However, this approach is not implemented in the PG vector store, and the community member is reluctant to read the documentation every time they want to retrieve data with BM25.

The community members discuss various approaches, such as retrieving the whole data and feeding it to BM25, and using the PG client to check the Postgres vector store documentation on how to retrieve all the documents. However, they note that the core code only requires fetching the relevant items based on the user's query, and it is not implemented to retrieve all the nodes from the database.

DDoğan

Hi everyone, how can i get the all nodes from en existing pg vector store to use it for bm25 retriever? I've already tried to get it with:

Plain Text

index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
nodes = index.docstore.docs.values()
print("nodes: ", nodes)

nodes: dict_values([])

It returns an empty dict even it has a value in the table

9 comments

WWhiteFang_Jr

Yeah the reason why it returns empty is because nodes are not present locally , they are stored in the vector stores.

You have the pg_client, Use that client to fetch all the data from vector store and then feed it to BM25

DDoğan

@WhiteFang_Jr Thank you so much, is there any examples or documents to get help? I'm new to llamaindex

DDoğan

I don't wnat to read the document every time i want to retrieve with bm25

WWhiteFang_Jr

Actually What I suggested in not implemented in PG vector store I think. 😅
But since you already have the client for PG vector store so its like DB connection client. You can use that and check on Postgress vector store official docs how they retrieve all the docs

WWhiteFang_Jr

Retrieve whole data and then feed it to BM25

DDoğan

Yeah i'm using pgvector

DDoğan

I don't know why is it too hard to get all nodes from the db 😄

WWhiteFang_Jr

Its not implemented as the core code only requires to fetch only the relatable items based on user query

DDoğan

@WhiteFang_Jr thank you

Add a reply

Find answers from the community

Hi everyone, how can i get the all nodes