Using a Local TEI Server for Reranking is Possible

Question

Is it possible to use a local TEI server for reranking?

sodeep · Answer

I just found this, but I'm not sure what to do with it. https://docs.llamaindex.ai/en/stable/api_reference/postprocessor/tei_rerank/

Logan M · Answer

Thats just api referencepip install llama-index-postprocessor-tei-rerankfrom llama_index.postprocessor.tei_rerank import TextEmbeddingInference as TEIR reranker = TEIR(...)

sodeep · Answer

Ah, thanks.

sodeep · Answer

You might want to update this issue, which makes people think it's still not possible. https://github.com/run-llama/llama_index/issues/9572

Logan M · Answer

2023 lol nice

Logan M · Answer

leaving a comment

sodeep · Answer

hmm, does LlamaIndex still not support TEI + gRPC?

Logan M · Answer

I guess not? Feel free to take a peek at the code or make a PR
https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/postprocessor/llama-index-postprocessor-tei-rerank/llama_index/postprocessor/tei_rerank/base.py

Logan M · Answer

I really haven't kept up with TEI

sodeep · Answer

TEI Reranker is working great! However, it fails when I try to combine it with the QueryFusionRetriever:'Input validation error: inputs must have less than 512 tokens. Given: 518'

sodeep · Answer

top_k = 25
vector_retriever = index.as_retriever(similarity_top_k=top_k)
bm25_retriever = BM25Retriever.from_defaults(docstore=index.docstore, similarity_top_k=top_k)
retriever = QueryFusionRetriever( [vector_retriever, bm25_retriever], retriever_weights=[0.6, 0.4], similarity_top_k=top_k, num_queries=1, mode='relative_score', verbose=True,
)

Logan M · Answer

set truncate_text=True in the TEI reranker

Logan M · Answer

seems like some text is just a touch too large

sodeep · Answer

I tried that. It's already set to True by default. No dice.

sodeep · Answer

Everything works fine if I forgo the QueryFusionRetriever.

Logan M · Answer

hmm, maybe TEI changed how that option works?

Logan M · Answer

it might not be sending it properly to TEI

sodeep · Answer

TEI rerank works great—if I replace the QueryFusionRetriever with a a regular ol' VectorStoreIndex...as_query_engine().

Logan M · Answer

I don't think the retriever should have any difference here lol its just coming down to how TEI is handling the truncation?

Unless the full traceback shows the error is happening somewhere else?

Also you might be more familiar than me, you can double check the API usage here
https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/postprocessor/llama-index-postprocessor-tei-rerank/llama_index/postprocessor/tei_rerank/base.py

sodeep · Answer

llama_index.postprocessor.tei_rerank.TextEmbeddingInference sets truncate_text on its parent class, BaseNodePostprocessor. I don't think TEI is respondible for truncation here.

sodeep · Answer

...but I'm wrong a lot

Logan M · Answer

Seems like it should be using it here. But it's not currently
https://github.com/run-llama/llama_index/blob/86208690c874b62b4d69ab2d6a44055387b7e597/llama-index-integrations/postprocessor/llama-index-postprocessor-tei-rerank/llama_index/postprocessor/tei_rerank/base.py#L85

sodeep · Answer

I'll try subclassing it.

sodeep · Answer

Thank you!

sodeep · Answer

Ah ha! The solution was to add --auto-truncate to my Docker command for the TEI reranker.

Logan M · Answer

thats so weird haha

Find answers from the community

Using a Local TEI Server for Reranking is Possible