Find answers from the community

Updated 10 months ago

Wondering if I can get some assistance

Wondering if I can get some assistance here.

I have the code below for an ingestion pipeline into a mongodb vector database, but getting some pydantic validation errors.


from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core import VectorStoreIndex
from llama_index.vector_stores.mongodb import MongoDBAtlasVectorSearch
vector_store = MongoDBAtlasVectorSearch(mongo_client, db_name=DB_NAME, collection_name=COLLECTION_NAME, index_name="vector_index")

pipeline = IngestionPipeline(
transformations=[
SentenceSplitter(chunk_size=500, chunk_overlap=0),
OpenAIEmbedding(model="text-embedding-3-small", dimensions=256),
],
vector_store=vector_store
)

Ingest directly into a vector db

pipeline.run(documents=llama_documents)


index = VectorStoreIndex.from_vector_store(vector_store)


Error:

---------------------------------------------------------------------------
ValidationError Traceback (most recent call last)
<ipython-input-12-6edc00a10dce> in <cell line: 11>()
9 print(type(vector_store))
10
---> 11 pipeline = IngestionPipeline(
12 transformations=[
13 SentenceSplitter(chunk_size=500, chunk_overlap=0),

1 frames
/usr/local/lib/python3.10/dist-packages/pydantic/v1/main.py in init(pydantic_self, **data)
339 values, fields_set, validation_error = validate_model(pydantic_self.class, data)
340 if validation_error:
--> 341 raise validation_error
342 try:
343 object_setattr(pydantic_self, 'dict', values)

ValidationError: 1 validation error for IngestionPipeline
vector_store
value is not a valid dict (type=type_error.dict)
L
r
12 comments
the atlas vector store needs to be updated with the new base class
Alright, I can see this.

I'll create a PR for the change in a minute
@Logan M I created a PR for this: https://github.com/run-llama/llama_index/pull/10698

I will be adding an example notebook shortly as well
No problem.

Do let me know what I can do to get this merged quickly.

Aiming to create a notebook and accompanying YT video this week.
left one comment! Then it should be good to go πŸ™‚
thanks, added the necessary changes
there was some issues with testing, but updated it now πŸ™‚

https://github.com/run-llama/llama_index/pull/10698
thanks for the support, looks like everything is all good.

I'll wait for it to be merged πŸ™‚
should be published now πŸ™‚
Add a reply
Sign up and join the conversation on Discord