Find answers from the community

Updated 2 months ago

Hello I am working on a project for

Hello, I am working on a project for document qa. But the thing is I need to do it with multilingual supported llm and embedding. So I am using hugging face embedding and llm but my outputs are complete nonsense. Can you help me?
from llama_index import VectorStoreIndex, ServiceContext
from llama_index import PromptHelper
from llama_index.llms import HuggingFaceLLM
from langchain.embeddings.huggingface import HuggingFaceBgeEmbeddings

embed_model = HuggingFaceBgeEmbeddings(model_name='sentence-transformers/all-MiniLM-L6-v2')

import torch

llm = HuggingFaceLLM(
tokenizer_name="bigscience/bloomz-1b7",
model_name="bigscience/bloomz-1b7",
model_kwargs={"load_in_8bit": True,"torch_dtype": torch.float16},
generate_kwargs={
"do_sample": True,
"top_k": 4,
"penalty_alpha": 0.6,
}
)

prompt_helper = PromptHelper(context_window=512, chunk_size_limit=200, num_output=100)
service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model, prompt_helper=prompt_helper)
index = VectorStoreIndex(nodes=nodes, service_context=service_context)
query_engine = index.as_query_engine()
L
E
6 comments
Couple of notes:
  • the embed class you are using is only for BGE embeddings (and the model you are specifying is also not great πŸ˜…). Use HuggingFaceEmbddings instead
  • The LLM you are using is not great tbh. It will probably have a super tough time following instructions and being used in llama-index. Is openai not an option here?
I am using these for a turkish document qa. So these were the options I found. πŸ™ƒ
So, finding the well-suited embedding and llm is my part but I have some questions. Sometimes, I see people use llm_predictor and sometimes just llm. Assume that I will always use embedding and llm from hugging face and use them for just very large scaled document qa. Do you have any tips or maybe examples?
Because I am kind of lost. I decided to use llama-index for my project because it is the best-looking framework for my need.
Yea I'm not really sure -- I haven't used bloom before. I would maybe test the LLM outside of llama-index to make sure it's actually capable of following instructions and whatnot. Otherwise, I would onsider using openai though if possible, the experience will be 10000x better
Add a reply
Sign up and join the conversation on Discord