Multimodal Integration with Cohere

theres a PR just opened to add colpali as a reranker

Adding it as an actual indexer is a lot harder due to handling multi-vector indexing, not there yet

is multi-vector indexing planned? does it perform much better than just using the cohere multimodal embeddings?

Some preliminary thoughts about how to do it, but nothing concrete.

Its extremely complex (and uses a lot more resources compared to dense embeddings)

Things that need refactoring to support it

• the node class assumes a single dense vector
• all our embedding model classes assume a single dense vector
• all our vector stores assume a single dense vector for retrieval

These are not easy things to fix

RE performance, I can't really comment. It largely depends on what your data looks like. IMO cohere multimodal should be fine in most cases

do you have benchmarks comparing cohere multimodal with full colipali with colqwen2 ?

https://huggingface.co/spaces/vidore/vidore-leaderboard
would be good to know how llama-index with recommended settings and best embeddings compares

could you consider running the vidore benchmark with your colipali reranker method?