Find answers from the community

Home
Members
Sridhar
S
Sridhar
Offline, last seen 3 months ago
Joined September 25, 2024
S
Sridhar
·

Please advise

Please advise
5 comments
L
S
anyone tried accessing llama_index.llms import Groq, i am not able to access this llm with
llama-index-llms-groq 0.1.3
llama-index-core 0.10.14
13 comments
S
M
L
Can any help to integrate


llm = Vllm(
model="mistralai/Mistral-7B-Instruct-v0.1",
dtype="float16",
tensor_parallel_size=4,
temperature=0,
max_new_tokens=100,
vllm_kwargs={
"swap_space": 1,
"gpu_memory_utilization": 0.5,
"max_model_len": 4096,
},
)
8 comments
L
S