Find answers from the community

Updated 3 weeks ago

Multimodal Integration with Cohere

I saw you now have multi-modal integration with cohere. Is there a colipali implementation in llamaindex too?
L
T
7 comments
theres a PR just opened to add colpali as a reranker
Adding it as an actual indexer is a lot harder due to handling multi-vector indexing, not there yet
is multi-vector indexing planned? does it perform much better than just using the cohere multimodal embeddings?
Some preliminary thoughts about how to do it, but nothing concrete.

Its extremely complex (and uses a lot more resources compared to dense embeddings)

Things that need refactoring to support it
  • the node class assumes a single dense vector
  • all our embedding model classes assume a single dense vector
  • all our vector stores assume a single dense vector for retrieval
These are not easy things to fix

RE performance, I can't really comment. It largely depends on what your data looks like. IMO cohere multimodal should be fine in most cases
do you have benchmarks comparing cohere multimodal with full colipali with colqwen2 ?
https://huggingface.co/spaces/vidore/vidore-leaderboard
would be good to know how llama-index with recommended settings and best embeddings compares
could you consider running the vidore benchmark with your colipali reranker method?
Add a reply
Sign up and join the conversation on Discord