How to make llamacpp model inference

Unless llamacpp was compiled for GPU, thats as fast as its getting

I don't have GPU. Iam running on cpu with 12 core 48 Gb Ram. Still it takes more than 2 mins for 7b model

that sounds about right. I have 16-core 48GB and I have had a similar experience on CPU 🙂

Tinyllama is giving in 40 seconds but output is not as expected for my Rag use case

Any best small language model to use with RAG

You should limit the llm calls and prepare retriving and prompting

Iam running tinyllama in llamacpp ( not pythonllamacpp its slow ) and i get like 10t/s on a 4gb vram built in mobile gpu and like 5-6t/s on gpu using like 4gb ram

How can it be done . Could you please guide me to examples .

I am using llamaindex LlamaCpp to load llm

https://docs.llamaindex.ai/en/latest/understanding/querying/querying.html

Configuring node post processors ?

Look into receivers also

And node size when idexing