Fred Bliss

Load in 8bit

hey all - is there a way to load huggingface models (local) in 8bit? i don't see the param in HuggingFaceLLMPredictor (it's a param in the transformers AutoModelForCausalLM)

1 comment

FFred Bliss

Community-Driven Building: How Open Sour...

Made a guide over the weekend and turned it into a blogpost on using open source LLMs /w autogptq + llamaindex for retrieval & tgo generate synthetic training data - figured i'd share in case it helps anyone. thanks for all your help unsticking me along the way. 🙂

https://fredbliss.substack.com/p/community-driven-building-how-open

1 comment

FFred Bliss

Has anyone used the ```data generator

Has anyone used the

Plain Text

data_generator.generate_questions_from_nodes()

pattern to generate synthetic question/answer pairs for finetuning datasets? Was working with some folks in another discord (local LLM focused) on generating synthetic instruction data for qlora/finetuning, and realized that all the data I need is indexed in a vector store already. This pattern works great for gpt4, hit or miss with local models - seems to mostly be issues between AutoGPTQ (4 bit quant library for GPU) and the transformer and/or HuggingFaceLLMPredictor in llama_index (borrowed from langchain?).

Working on a solution across a few different threads, just curious if anyone went down this path yet.

edit: but by using this + https://github.com/OpenAccess-AI-Collective/axolotl for prompt strategies for converting to jsonl formats for a given instruction set, it's a pretty great solution. Just costly to use gpt4 to generate them. 🙂

9 comments

FFred Bliss

where does Document fit into this

where does Document fit into this picture?

2 comments

FFred Bliss

Aside from making the storage larger and

Aside from making the storage larger and slower to perform, I'm not seeing an accuracy change up or down - but I'm also lacking metrics to say definitively, or if there's under-the-hood processes that would take advantage of both sources of data in different ways.

6 comments

FFred Bliss

```

Plain Text

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 6
      1 from llama_index import (
      2     SimpleDirectoryReader,
      3     ServiceContext,
      4     LLMPredictor
      5 )
----> 6 from llama_index.experimental.evaporate import EvaporateExtractor
      7 from langchain.llms.openai import OpenAIChat, OpenAI
      8 import requests

ModuleNotFoundError: No module named 'llama_index.experimental'

5 comments

Find answers from the community

Load in 8bit

Community-Driven Building: How Open Sour...

Has anyone used the ```data generator

where does Document fit into this

Aside from making the storage larger and

```