I am trying to query custom documents

ppraveenks30#120294

I am trying to query custom documents using LlamaIndex and using MongoDB altas for backend. below is the code for the same. The code is working fine even I can see the documents are created successfully in MongoDB in default_collection under default_db but I am getting None on the response of my query. Could you please let me know what is missing and what needs to be added to get the response .

import pymongo
import openai
import os
from llama_index import VectorStoreIndex, StorageContext, SimpleDirectoryReader
from llama_index.vector_stores.mongodb import MongoDBAtlasVectorSearch
from llama_index.storage.storage_context import StorageContext


openai.api_key = os.getenv("OPENAI_API_KEY")

# setup mongo connection
mongo_uri = os.environ(mongo_uri)

# setup client
mongodb_client = pymongo.MongoClient(mongo_uri)

if mongodb_client :
    print("Connection is successful!")
else:
    print("Connection is not successful!")

# setup store
store = MongoDBAtlasVectorSearch(mongodb_client)
# print(store)

# setup storage context
storage_context = StorageContext.from_defaults(
    vector_store= store
)

documents = SimpleDirectoryReader("input/text").load_data()
index = VectorStoreIndex.from_documents(documents, storage_context= storage_context)
print(index)

# ask query 
query_engine = index.as_query_engine()
response = query_engine.query("What did author  work on after college?")
print(response)

16 comments

WWhiteFang_Jr

Code looks fine. Could you try checking

When you load documents from the folder, If it is not empty.
Once you have the documents, Check the length of it by doing print(len(documents))

If the length is more than or equal to 1 you can try chekcing the content with

Plain Text

for doc in documents:
  print(doc.text)

ppraveenks30#120294

Yes. I can see the output.

print(len(documents) --> 3
I can also see the output of all files when printing using print(doc.txt)

ppraveenks30#120294

Everything seems fine till index creation but while asking any query, it's returning None.

ppraveenks30#120294

It is also got loaded into MongoDB Atlas :

Attachment

WWhiteFang_Jr

Can you try running this piece of code and see if you get any response or not

Plain Text

from llama_index import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader('input/text').load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What did author  work on after college?")
print(response)

ppraveenks30#120294

Yes, I am getting a response.

ppraveenks30#120294

Also, I am getting a response from below code. Please do note that I have commented out MongoDBAtlasVectorSearch.

# setup store
# store = MongoDBAtlasVectorSearch(mongodb_client)
# print(store)
# setup index
storage_context = StorageContext.from_defaults(  docstore=MongoDocumentStore.from_uri(uri=mongo_uri),
index_store=MongoIndexStore.from_uri(uri=mongo_uri),
)

WWhiteFang_Jr

Okay one last thing
Run by uncommenting store = MongoDBAtlasVectorSearch(mongodb_client) part and when index is created do print print(index.docstore.docs)

I have not tried MongoDBAtlasVectorSearch, So Not sure why this part is not working. @Logan M Could you take a look into this

LLogan M

@praveenks30#120294 @WhiteFang_Jr atlasdb is so annoying to setup

Once you create the index in mongodb, you have to do a manual step in the GUI to turn it into a searchable vector index. Last time i checked, there was no way to do this with code, which is just really janky

I can't remember the exact steps off the top of my head, but you have to create a "search index" in the GUI, on top of the DB you created. When setting up the DB, there should be a JSON config editor, and you have to paste in some JSON. I figured this out at some point by stitching together langchain docs and random google search results LOL

Should really record this process somewhere. But also it's so janky that it's hard to recommend anyways
https://python.langchain.com/docs/integrations/vectorstores/mongodb_atlas

ppraveenks30#120294

Thanks for the direction, Logan ! Atleast it gives some direction and clarity !

aash7

Hi Praveen, were you able to resolve this issue? Even I am getting None as output. I have created Search Index in Mongodb Atlas but still the same issue.

LLogan M

@ash7 when you created the index, did you paste in that json config?

LLogan M

@ash7 check out this message thread https://discord.com/channels/1059199217496772688/1143899419578728528/1143901699006791720

aash7

@Logan M Thank you for your response. I got it working by keeping the index name same in mongodb GUI and index name it is in python. By defualt it uses "default" as index name. Now I am struggling to change the default names for mongdo db and collection. It uses "default_db" and "default_collection" automatically. Kindly help if you have something on how to change these names?

Currenty code snippet looks like this...

import pymongo
import openai
import os
from llama_index import VectorStoreIndex, StorageContext, SimpleDirectoryReader
from llama_index.vector_stores.mongodb import MongoDBAtlasVectorSearch
from llama_index import LLMPredictor, ServiceContext
from langchain.chat_models import ChatOpenAI

def construct_index(email):
openai.api_key = os.getenv("OPENAI_API_KEY")

# setup mongo connection
mongo_uri = os.getenv("URI")

# setup client
mongodb_client = pymongo.MongoClient(mongo_uri)

llm_predictor = LLMPredictor(llm=ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo"))

service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)

if mongodb_client:
print("Connection is successful!")
else:
print("Connection is not successful!")

# setup store
store = MongoDBAtlasVectorSearch(mongodb_client)

# setup storage context
storage_context = StorageContext.from_defaults(
vector_store=store
)
data = "PATH_TO_DOCS"
# data = "/app/data/docs/"
documents = SimpleDirectoryReader(data).load_data()
index = VectorStoreIndex.from_documents(documents,
storage_context=storage_context,
service_context=service_context)
return index

LLogan M

You can change the names in the contrsuctor for MongoDBAtlasVectorSearch

Attachment

aash7

Thank you @Logan M. It is completed now.

Add a reply

Find answers from the community

I am trying to query custom documents