Find answers from the community

Updated 2 months ago

Err weird question hoping anyone else is

Err weird question hoping anyone else is/has seeing this: Has anyone else's retriever suddenly become very dumb over night? I'm using OpenAI's Davinci model for embedding and I swear yesterday it was doing great. Today, I query something I've queried before and it returns the worst nodes. I'm so confused.

14 comments

LLogan M

have not seen this 👀 Although I know davinci is being deprecated by openai -- could be something weird going on under the hood on their end

PPocketColin

that would be weird. They just added Davinci as an embedding model though I thought...

PPocketColin

I've been meaning to mess around with text-embedding-ada-002 anyway so I'll give that a shot.

LLogan M

ada-002 is about 500x cheaper it looks like lol

LLogan M

maybe a good call

PPocketColin

yeah haha so much cheaper and apparently better but idk I tested it before and had mixed results.

PPocketColin

I spent $440 on embeddings over the last week 😆 hoping my company doesn't get upset about the big charge 😬

LLogan M

🤑 💸

PPocketColin

Looks like that weird dumbing down of Davinci was a blessing in disguise! ada-2 is working great so far!

PPocketColin

Ugh ok after a ton of testing Ada is only producing results with ~80% accuracy while Davinci had 92%. Today I tried Davinci again and sure enough it's back up... I wonder what they broke yesterday lol

PPocketColin

any other embedding models/AI companies I should look at?

LLogan M

hmm. Cohere-Rerank is very popular (but it's a paid API for reranking, that we support)

I know JinaAI is about to launch their V2 embeddings (but they aren't out yet)

The rest is just stuff from this leaderboard -- https://huggingface.co/spaces/mteb/leaderboard -- I would maybe give bge-large-en-v1.5 a try again, but set your chunk size to 512

PPocketColin

I set the chunk size to 512 and it didn't make any difference 😿

LLogan M

Hm, weird 🤷‍♂️

Add a reply