Hey i m a big beginner in AI that s my

VVaylonn

Hey, i'm a big beginner in AI (that's my first project). I would have like to know some answers on my code that I encounter.
I want to implement something that take info in my files using local LLMs like vicuna or alpaca, instead of open AI

I know that the format of the code should look like that:
for exemple with PDFs:

libraries

from llama_index import GPTVectorStoreIndex, LLMPredictor, download_loader
from pathlib import Path
from llama_index import download_loader

-connexion to LLM (open AI or customs) (dont know how to do this part cause i cant find any exemple, everything is different)

then

"plugins" from llamahub.ai to give access to documents

PDF_NAME = '...'

file = requests.get('web_adress_to_pdf'.format(PDF_NAME), stream=True)
with open(PDF_NAME, 'wb') as location:
shutil.copyfileobj(file.raw, location)
PDFReader = download_loader("PDFReader")

loader = PDFReader()
documents = loader.load_data(file=Path('./article.pdf'))
index = GPTVectorStoreIndex(documents, llm_predictor=llm_predictor)

prompt + answers

response = index.query("prompt")
print(response)

If you know how to solve this, i would like to know ! 🙂

21 comments

LLogan M

In the newest versions of llama index, you can load any LLM from huggingface

https://gpt-index.readthedocs.io/en/latest/how_to/customization/custom_llms.html#example-using-a-huggingface-llm

If its not on huggingface, there's also another example just below that

LLogan M

This will run all the LLMs locally

LLogan M

You can also use huggingface for a local embeddings model
https://gpt-index.readthedocs.io/en/latest/how_to/customization/embeddings.html#custom-embeddings

LLogan M

ah, I should have scrolled further, you've read all this haha

VVaylonn

yeah ! 🙂

VVaylonn

i guess my problem is that i don't know how to code the link between llamaindex and my llm locally as you said on the blog to somewhat "take the prompt give it to the llm" and then take back what he said

LLogan M

Yea so if you follow the notebooks in the docs, they should cover everything

You create the HuggingFaceLLMPredictor, attach it to the service context, and then build your index with that service context

Then you can use the query engine from the index, which will embed your query (using openai, or a local embedding model if you also set that up), and use that query embedding to fetch relevant text. The LLM reads that text and then answers your query 💪

VVaylonn

I’ll try to do that ! Thank’s for your help and I’ll let you know if I find any issues

oomari5252

If anyone else is curious, here’s how I got it working https://youtu.be/kZCEj8GOOnA

VVaylonn

oooh, that's sounds good

VVaylonn

Same issue here than the man in your comments

Attachment

VVaylonn

else, everything was working well

Attachment

LLogan M

You'll want to set an embed model as well, the llm is just for generating text

https://gpt-index.readthedocs.io/en/latest/how_to/customization/embeddings.html#custom-embeddings

VVaylonn

everything went well until when i queried.. @Logan M

Attachment

VVaylonn

Attachment

VVaylonn

here is the rest of the modified code

Attachment

LLogan M

A classic lol. tbh the model seems extremely constrained (only 240 max input size is teeny tiny, OpenAI is 4096 in comparison)

You can likely avoid this error by setting max_chunk_overlap=-1000 (a weird hack, but it works). But I don't think you'll get any useful results due to the input size 😅

VVaylonn

yeah, that was depending on the model actually, planning to change it to a bigger one like vicuna or smth

VVaylonn

(a weird hack, but it works) 🤣

VVaylonn

thank's a lot for all you're help and evrything you do for the community ! lot of love

LLogan M

Happy to help! 💪

Add a reply

Find answers from the community

Hey i m a big beginner in AI that s my