Howdy folks Is it possible to use GPT J

If you have a model hosted somewhere you can get your index to use that if thats what you mean

You need a model for inference somewhere

Thanks! I'm trying to do some napkin estimations on deployment costs. How much would it cost to index 1M tokens on OpenAI ?

I think making embeddings is just cost-per-token, and once it's in a vector database retrieval is just whatever that costs

however worth noting that gpt_index retrieves from the vector store and uses an LLM call to answer your query so if you want to be completely "offline" you'll have to work with the store directly (not a huge deal)

Although, now that I think about it, I think inserting into the SimpleVectorStore is an offline process so there might be a local model doing it in gpt_index...

Jerry would know more he'll probably chime in eventually

Got it. I don't quite yet have a good understanding of how the vectors are retrieved from the OpenAI service.

Also: how does gpt-index relate to model fine-tuning?

This might be helpful context: https://www.youtube.com/watch?v=YkK5IKgxp-c

TLDR model magic lol

Also: how does gpt-index relate to model fine-tuning?

Kind of orthogonal, gpt_index is used so one doesnt need to fine-tune a model over a dataset. It facilitates retrieval of a subset of a document for use in a prompt.

we just langchain's llm model under the hood, and i think it should be possible to specify the LLM model as gpt-j? https://discord.com/channels/1059199217496772688/1059201661417037995/1060680748316766279

see https://gpt-index.readthedocs.io/en/latest/how_to/custom_llms.html for how to customize the langchain llm and use it in gpt index

GPT-J TPUs will set you back a lot: https://devforth.io/blog/gpt-j-is-a-self-hosted-open-source-analog-of-gpt-3-how-to-run-in-docker/

I was actually looking at using NLPCloud instead.