Hi folks first of all thanks for this

SSergio Casero

Hi folks, first of all, thanks for this awesome job

I'm trying to estimate the costs of the "training", the use case is the following: I have lot of pdfs and I want to integrate them with LLM. By using the MockLLMPredictor, I get the following info attached (all of them based on SimpleDirectoryReader, same dir), the question is... does these values have sense?, the "per query" it's obviously a query estimation based on 5 five queries made with Mocks.

Attachment

19 comments

LLogan M

Hmm, I get slightly different costs for some reason

The Cost column includes the build tokens + 50 queries right?

Attachment

SSergio Casero

Yep

SSergio Casero

But the build cost is just 1 time, no?

LLogan M

oh whoops lol you are right

SSergio Casero

This is the formula (IMHO)

SSergio Casero

Attachment

LLogan M

yea you got, I was multiplying the build for every query lol

SSergio Casero

hahaha, noooo I don't want to pay extra cost hahaha

SSergio Casero

Just FYI this is just the 10% of the docs hahahaha

LLogan M

Concerning the cost these 3 indexes are definitely the most expensive 🙂

You might also be interested in making a vector index for each PDF (if they are all pretty separate topics) and then combining them all with a top level tree or keyword index, instead of a single vector index
https://gpt-index.readthedocs.io/en/latest/how_to/index_structs/composability.html

Lots of small experiments you could run to try and get the best results.

SSergio Casero

I was thinking on splitting by year

LLogan M

Oh boy hahah

If you end up using a vector index, look into using pinecone or qdrant (or similar) to store the vectors. Otherwise your computer will hate you for loading so much into memory

SSergio Casero

But jmmm, it looks pretty good

SSergio Casero

16TB is enough?

SSergio Casero

Or you mean RAM

LLogan M

Yea RAM

With the GPTSimpleVectorIndex, it stores the embedding for each document chunk in memory. Good for testing and when the index is smaller

But dedicated vector stores have this all optimized

SSergio Casero

Cool thanks, will ty to learn this behaviour

SSergio Casero

Hope to have something to show in the #😎app-showcase soon 🙂

SSergio Casero

~~Jmmm I think I'm missing something, if I try to create the index with~~ GPTSimpleVectorIndex ~~and mock predictor but I get~~ ~~"quota api excc" errors, but I'm using the~~..~~. mock, right~~? Forget it, some errors in the code

Add a reply

Find answers from the community

Hi folks first of all thanks for this