What are the tradeoffs to doing rankings

i don't see much tradeoff except for additional latency or bottlenecks if you're doing it outside the vectordb. There is also the case where the vectordb ranking implementation isn't that good (no specific cases in mind).
Which vectordb are you looking at?

Mostly at Qdrant and Vespa

Due their existing or soon to be support for high dimensional vectors and multi-vector embedding models

Pick your vector embedding model needs and then your vector db! bge-m3 support is quite good for in these VectorDBs

Vespa rocks. It's my favorite when i have a serious enterprise use case

cause Vespa in essence is an enterprise level Information retrieval engine, which "also" now has support for similarity search ,etc.. highly customizable and they adapt new frameworks and methods rather really quickly

Qdrant really good performance and a small memory footprint.