Find answers from the community

Updated 3 months ago

Llm

Hi,
I am trying to do Q&A on docs. I tried the below code but I am getting error. Can someone please help me fixing the error.
Plain Text
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
context_window = 1024
num_output = 256
model_name = 'GPT2'
embedding_model = '/home/Coding/Coding Stuff/AI Models/GTE-Base/'
embding = HuggingFaceEmbedding(model_name=embedding_model, device=device)

class Our_own_llm(CustomLLM):
    @property
    def metadata(self) -> LLMMetadata:
        '''Get LLM metadata'''
        return LLMMetadata(context_window=context_window, num_output=num_output, model_name=model_name)
    @llm_completion_callback()
    def complete(self, query:str, max_length=1024, temperature=0.5, **kwargs:Any) -> CompletionResponse:
        '''Complete prompt'''
        api_url = "http://127.0.0.1:5000/own_gpt"
        data = {"context": None, "question": query, "max_length": max_length, "temperature": temperature}
        response = requests.post(url = api_url, json = data)
        result = response.text
        return CompletionResponse(text=result)
    @llm_completion_callback()
    def stream_complete(self, context:str, query:str, **kwargs:Any) -> CompletionResponseGen:
        raise NotImplementedError()
    
def context_service():
    llm = Our_own_llm()
    service_context = ServiceContext.from_defaults(llm=llm, context_window=context_window, num_output=num_output, embed_model=embding)
    return service_context

file = '/home/laxmidhar/Coding Stuff/Data_House/interview questions.docx'
query = 'what is selection bias?'

service_context = context_service()
docx_document = DocxReader()
data = docx_document.load_data(file=Path(file))
index = ListIndex.from_documents(data, service_context=service_context, embd_model = embding)
query_engine = index.as_query_engine()
response = query_engine.query(query)

The error details in the txt file.
L
l
6 comments
You have an extremely small context window size (1024 tokens) and also specifying 256 output tokens.

This means the largest input to the llm can be 1024 minus 256 tokens.

Tbh the chunking algorithms really struggle with these smaller sizes.

I recommend just... not using gpt2 lol (it's also not even worth the time, the outputs won't be good)
Okay.
Itried Q&A with image with below code in two different system but it is not working.
Can you please guide me.
Plain Text
Image_document = image_reader.ImageReader()
        data = Image_document.load_data(file=Path(file))
        index = ListIndex.from_documents(data, service_context=service_context, embd_model = embding)
        # index_1 = VectorStoreIndex.from_documents(data, service_context=service_context)
        return query_response(index, query)
Having you seen our multi modal guides?
Thanks for the link.
Let me go through the documentation. If i have any doubt will post here.
Hi @Logan M
I follow the below link for Q&A with image but i am getting erro
https://docs.llamaindex.ai/en/stable/examples/multi_modal/ollama_cookbook.html#structured-data-extraction-from-images
This error from my pc - ConnectError: [Errno 111] Connection refused
This is from office pc - AttributeError: 'Document' object has no attribute 'image'
The other approaches which is in the documentation we can't use those for the comercial purpose. So is there any method or approach is there where we can do Q&A or querrying the image.

Thanking you in advance.
Add a reply
Sign up and join the conversation on Discord