Find answers from the community

Updated 2 months ago

remove model from GPU

Hello, I am trying to benchmark performance of some open source models on a custom datasets using HuggingFaceLLM method in llamaindex. I want to test the models sequentially. How do I remove a particular model from the GPU before moving the next model to the GPU?
r
D
2 comments
I think following code should work.

Plain Text
del model
torch.cuda.empty_cache() 
I tried it before but it didn’t work
Add a reply
Sign up and join the conversation on Discord