I am using llama index to build a local

A <1GB model will never work well tbh. It's pretty much impossible.

I've found an LLM needs at least 13 billion parameters to be useful. 1GB is like 1B parameters or less 😅

The closest thing I've seen is this project, but I haven't tried it out yet
https://github.com/aitomatic/openssm

thanks for pointing the direction. I'll take a look.