MongoDB Atlas

I think i might have misunderstood what is mongodb created for

My use case is that i want to stores my indexes, indexed in a vector store format with GPTVectorStoreIndex, and query over them without having to load any documents or index any documents at first.

Just using .json with an embedded documents in it

Should I be able to do that with MongoDB Atlas ? If yes, how ?

And if not, is my tool to do that would be a vector database such as Pinecone ?

You should be able to do it with at atlas or pinecone 👀

Just need to setup the vector store and storage context and create the index

Then to "load" the index, setup the vector store again and do something like

index = VectorStoreIndex.from_vector_store(vector_store)

I guess that's what I'm doing here:

But then I get this error: File "c:\Projets\IA Chat Local\Sources\AzureOpenAI\app copy.py", line 203, in get_json index = VectorStoreIndex.from_vector_store(vector_store="DTU") File "C:\Users\sxd-i\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\indices\vector_store\base.py", line 60, in from_vector_store if not vector_store.stores_text: AttributeError: 'str' object has no attribute 'stores_text'

And I instanciated my db like that:

I think I might have errors on: 1- the format of the index in Mongo
2 - The calling part of the index

Ah, looks like a small bug, the vector store is missing that attribute 😢

Crap !

But still, is the index instanciated in the right way in Mongo ?

I can't find/understand how llama_index processes it when using it for the query so idk wich part he's taking

oh wait, I read this closer, you had an error in your code, one sec!

index = VectorStoreIndex.from_vector_store(store) <- should be that instead of passing in a string

yeah i had this result before, it fixed it !

Where i have the problem now

is that as a result, i should have an entire json with some response

I guess that when doing that, it doesnt take any info from the index, and thus can't query over it

🤔 it should work... one sec. I need to test this part of the code for another PR, so may as well test it for this too lol

I should have something like that

I can give you my code if you want

nah its all good. Just figuring out how to create a mongodb instance lol

Does it work if you query right after creating the initial index? I can't even get an initial response 😦

I can see it inserted everything though

I don't even create it, i import it trough my terminal

like the files

that's why i'm not sure of the format

mongoimport --uri mongodb+srv://xxxxxxx:xxxxxx@indexatlasdb.vwrbmy3.mongodb.net/DTU --collection index_store --type json --file index_store.json

ohhhhhhh you uploaded the entire index into mongodb? I thought you just used the vector store 😅

Using just the vector store takes care of everything, no need to worry about index_store or docstore

I'm on a slightly different unreleased version of llama-index, so it will look slightly different for you, but every entry in the vector store includes the node data

I tried to do it this way but i didnt managed to sooooo

whent for the hard one x)

Would mind to show me the part of the code doing it ?

But also, I'm struggling to get an answer for a query lol from both the initial index and the loaded index

trying to figure out why the mongo query returns none 😦

But also, this is my first time looking at mongo db code hahaha

Hmm I think I missed a step, I think I need to configure a mapping/embedding field in my db first

it doesnt even create the index for me :/// It created it one time, then I deleted it to be sure

and now nothing

and still get the same error as before

Ok, when querying like this:

I have an output as Null:

because i have no index at all or i "load" nothing

and with this:

No index is created, and i get a program error

even though it created an index one time on the first time

aaaaaah that's pisses me off, i change nothing and it doesnt want to be created anymore

I got it to work!

ok, so the screenshot I shared before is correct (for the creation and loading)

but in your mongodb portal, you need to setup a search index. Use the JSOn editor and do something like this

wait a few minutes, and then queries start working

this is not well documented at all LOL

I got the setup process from the langchain docs actually. But even those instructions were bad https://python.langchain.com/docs/modules/data_connection/vectorstores/integrations/mongodb_atlas_vector_search

let me know if you have troubles creating the search index. Once I did that in their UI, everything worked 🙂

I will try it tomorrow and I let you know !

Okkkkkkk

i'm stoopid

from_documents for the indexation

from_vector_storefor the retrieval

long time no see this error

i had it a way ago

then it disappeared idk how

and it came back

with this technique

Ok ! I got it all working !!!

few updates for your knowledge:

1 - If you want to add a document to an existing index, you don't need to recreate the index from 0 with all the base documents. You can just add the new document to the end of the existing index

2 - when you create the serach index, do not change the name "default" here either it'll not work

3 - you can query by two different manners:

3.1 - if you want to query over an existing index named db_name for exemple or dont add it if you want to use the default one

But also

3.2 - this way, while adding new documents over the existing index, you can query too with the new updated index

and the index is created also with this technique (if not created before) with the db_name argument of your choice

as a str

Et voilà !

Btw you can have also different indexes:

and the search engines linked to every one of them:

Ayyyy thanks for the detailed stuff! Amazing! 👏 👏