Hi - having some trouble with local

At a glance

The community member is having trouble loading local embeddings (huggingface, onyx, langchang) in their LlamaCPP setup. They get an error related to the cublas64_11.dll file not being found. Other community members chime in, with one noting that they were never able to get LlamaCPP working on their Windows machine. However, another community member provides an update, stating that they were able to get it working by loading the embed model first, though they are unsure if it's a DLL loading issue or a version mismatch.

YYarHarHAR

Hi - having some trouble with local embeddings:

Following the basic getting started tutorial
Installed LlamaCPP via the tutorial - and it works
Installed SentenceTransformers and verified that works
Got the basic prompt completion working in the tutorial

When I try to load any kind of local embedding (huggingface, onyx, langchang), I get the following error (in thread):

4 comments

YYarHarHAR

Plain Text

ex. embed_model = OptimumEmbedding(folder_name="./bge_onnx")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\yarha\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\llama_index\embeddings\huggingface_optimum.py", line 38, in __init__
    from optimum.onnxruntime import ORTModelForFeatureExtraction
  File "C:\Users\yarha\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\optimum\onnxruntime\__init__.py", line 18, in <module>
    from ..utils import is_diffusers_available
  File "C:\Users\yarha\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\optimum\utils\__init__.py", line 44, in <module>
    from .input_generators import (
  File "C:\Users\yarha\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\optimum\utils\input_generators.py", line 29, in <module>
    import torch
  File "C:\Users\yarha\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\__init__.py", line 122, in <module>
    raise err
OSError: [WinError 127] The specified procedure could not be found. Error loading "C:\Users\yarha\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\lib\cublas64_11.dll" or one of its dependencies.

Any idea what is happening here? cuBlas is already being loaded and used successfully with the LLamaCPP

LLogan M

Oh no windows 😪

LLogan M

I was never able to get llamacpp working on my windows machine

YYarHarHAR

Update - got this working by loading the embed model first. Don't know if it's a trying-to-load the samel dll twice thing or maybe a slightly different version of cublas, but llamacpp handles that situation better apparently

Add a reply

Find answers from the community

Hi - having some trouble with local