Also would it be possible to use any of

llama.cpp needs to be compiled for your GPU -- see full instructions on their readme

https://github.com/abetlen/llama-cpp-python

Then, you can set num_gpu_layers to something other than zero to utilize the GPU. -1 will offload all layers to your GPU (this assumes you have enough VRAM though)
https://gpt-index.readthedocs.io/en/stable/examples/llm/llama_2_llama_cpp.html#setup-llm