Find answers from the community

Updated 2 months ago

remove model from GPU

Hello, I am trying to benchmark performance of some open source models on a custom datasets using HuggingFaceLLM method in llamaindex. I want to test the models sequentially. How do I remove a particular model from the GPU before moving the next model to the GPU?

2 comments

rravitheja

I think following code should work.

Plain Text

del model
torch.cuda.empty_cache()

DDavid Akinpelu

I tried it before but it didn’t work

Add a reply