Hi everyone, I am using
RetrieverEvaluator
for a document in Vietnamese with a 4-bit quantized version of Llama-3-8B-Instruct:
llm = HuggingFaceLLM(
model_name="meta-llama/Meta-Llama-3-8B-Instruct",
system_prompt=system_prompt,
query_wrapper_prompt=query_wrapper_prompt,
context_window=8192,
max_new_tokens=256,
model_kwargs={
"token": hf_token,
"torch_dtype": torch.bfloat16, # comment this line and uncomment below to use 4bit
"quantization_config": quantization_config
},
generate_kwargs={
"do_sample": True,
"temperature": 0.1,
"top_p": 0.3,
},
tokenizer_name="meta-llama/Meta-Llama-3-8B-Instruct",
tokenizer_kwargs={"token": hf_token},
stopping_ids=stopping_ids,
)
My run is similar to this example in
https://docs.llamaindex.ai/en/stable/examples/evaluation/retrieval/retriever_eval/, however when I look into the dataset from eval_dataset.json, all questions have already been translated to English though. Do you have any insights about what happened, and what could I make to make sure that the questions extracted from the dataset are in Vietnamese (or in any other languages apart from English)?