LlamaIndex
Log in

Find answers from the community

Home
Members
BenBot
B
BenBot
Offline, last seen 3 months ago
Joined September 25, 2024
BBenBot
·

Alright got another one for ya ll

Alright, got another one for ya'll.

I'm trying to use vllm to run a model. vllm provides an openai-compatible API, BUT i need to use a custom model name.
I can't seem to find out how.

It seems like it'll always throw the error at line 188 in openai_utils.py.

Is that right? Is there now way to put in custom model names? If not, i'm probably going to try opening a PR for that
6 comments
B
W
BBenBot
·

Hey again I m running into some issues

Hey again, I'm running into some issues running a GPTQ model locally with llama-index.

I'm seeing a CUBLAS_STATUS_NOT_SUPPORTED error when trying to make a query. It's really weird because i'm able to use transformers to run this same GPTQ model without llama-index, but running it within llama-index is giving me this error.

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmStridedBatchedExFix( handle, opa, opb, m, n, k, (void*)(&falpha), a, CUDA_R_16F, lda, stridea, b, CUDA_R_16F, ldb, strideb, (void*)(&fbeta), c, CUDA_R_16F, ldc, stridec, num_batches, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)

The only options i gave to HuggingFaceLLM is model_name and device_map="auto"

Any idea what steps I should take?
6 comments
L
B
BBenBot
·

Hey all i m trying to use llama index

Hey all, i'm trying to use llama_index from main since it contains a fix for a breaking llama-cpp change, but I can't seem to figure out how.
I set up poetry to pull from the llama_index github, but when I try importing ServiceContext I'm met with this error

Plain Text
>>> from llama_index import ServiceContext
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/benbot/.cache/pypoetry/virtualenvs/llamaindex-learn-LAh3hCYO-py3.11/lib/python3.11/site-packages/llama_index/__init__.py", line 41, in <module>
    from llama_index.indices.struct_store.sql import (
  File "/home/benbot/.cache/pypoetry/virtualenvs/llamaindex-learn-LAh3hCYO-py3.11/lib/python3.11/site-packages/llama_index/indices/struct_store/__init__.py", line 5, in <module>
    from llama_index.indices.struct_store.sql import (
  File "/home/benbot/.cache/pypoetry/virtualenvs/llamaindex-learn-LAh3hCYO-py3.11/lib/python3.11/site-packages/llama_index/indices/struct_store/sql.py", line 9, in <module>
    from llama_index.indices.common.struct_store.sql import SQLStructDatapointExtractor
  File "/home/benbot/.cache/pypoetry/virtualenvs/llamaindex-learn-LAh3hCYO-py3.11/lib/python3.11/site-packages/llama_index/indices/common/struct_store/sql.py", line 8, in <module>
    from llama_index.indices.common.struct_store.base import (
  File "/home/benbot/.cache/pypoetry/virtualenvs/llamaindex-learn-LAh3hCYO-py3.11/lib/python3.11/site-packages/llama_index/indices/common/struct_store/base.py", line 10, in <module>
    from llama_index.utilities.sql_wrapper import SQLDatabase
ModuleNotFoundError: No module named 'llama_index.utilities'


I'm not really an expert in the python packaging system and I can't find any instructions for running llama_index from main, so if anyone knows how it'd be a big help 😅
21 comments
B
L
BBenBot
·

Embeddings

FWIW it also seems to happen when i try to use a HuggingFaceEmbedding too, which is kind of weird
1 comment
L
BBenBot
·

Llamacpp

Has anyone run into this error before when trying to use a local embedding model in a service context?

AttributeError: 'Llama' object has no attribute 'params'
4 comments
B
L