Find answers from the community

Updated 3 months ago

Quick question. My ingestion pipeline

Quick question. My ingestion pipeline works just fine for building my vector store. But, as my postgres DB expanded, I needed to embed a few other fields not related to the original ingestion. Would those here, hand-roll a new embedding table? Just create a new VectorStoreIndex? I definitely don't want to add the new embeddings to the original. Any perspectives welcome.

6 comments

LLogan M

I thiiiink you could just run the SQL queries with sqlalchemy to add columns to the existing table?

JJasonV

So, just place a description_embedding: Optional[List[float]] = Field(None, sa_type=ARRAY(Float)) to the original table?

JJasonV

I thought it cleaner to separate out the embeddings into new tables. But, .... maybe that's overkill.

LLogan M

Yea it kind of depends how much work you want to make for yourself lol

JJasonV

Your idea seems like the simplest. Populating that column would be pretty straightforward, too.

I like to strike a balance of get it done vs do it right. 🙂

Thanks.

So, I have description a str I can easily embed. But, I also have description_keywords I want embedded. That seems like overkill. 1k vectors for 3,000 terms. Hm.

JJasonV

Need to think a little more.

Add a reply