I'm trying to run llama index with llama

CCipher Studies

I'm trying to run llama index with llama cpp by following the installation docs but inside a docker container.

DOCKERFILE

Plain Text

# Use the official Python image for Python 3.11
FROM python:3.11

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# ARG FORCE_CMAKE=1

# ARG CMAKE_ARGS="-DLLAMA_CUBLAS=on"


# Install project dependencies

RUN CMAKE_ARGS="-DLLAMA_CUBLAS=on" python -m pip install -r requirements.txt

# Command to run the server
CMD ["python", "./server.py"]

Problem:
For some reason, the env variables in the llama cpp docs do not work as expected in a docker container.

Current behaviour: BLAS= 0 (llm using CPU)

Expected behaviour: BLAS= 1 (llm using GPU)

Attachment

9 comments

LLogan M

Is your GPU even visible inside the docker container?

CCipher Studies

I ran nvidia smi inside the container and yes it's visible there.

LLogan M

Not entirely sure then -- I've never tried running llama.cpp through docker

CCipher Studies

ah that sucks, could you recommend any other discord servers where I can get help regarding this?

CCipher Studies

to anyone who stumbles upon this thread:
I solved this using a cuda image

updated dockerfile:

Plain Text

FROM nvidia/cuda:12.3.0-devel-ubuntu22.04

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install Python and pip
RUN apt-get update && apt-get install -y python3 python3-pip

# Set environment variable
ENV CMAKE_ARGS="-DLLAMA_CUBLAS=ON"

# Install Python dependencies
RUN pip install --no-cache-dir --upgrade pip && \
    pip install -r requirements.txt --no-cache-dir

# Command to run the server
CMD ["python3", "./server.py"]

CCipher Studies

@Logan M tagging you to let you know.

CCipher Studies

https://stackoverflow.com/questions/77534658/no-gpu-support-while-running-llama-cpp-python-inside-a-docker-container

LLogan M

nice!

CCipher Studies

had to ask on stack overflow myself since there was no other question lol.

Add a reply

Find answers from the community

I'm trying to run llama index with llama