losing my hair over this I ve installed

At a glance

losing my hair over this, I've installed a fresh ubuntu 20.04, installed python3.11, setup a venv within my flask app running off gunicorn within the app to run python3.11 .. I have llama index working but whenever i try upload a new file type i get a modulenotfound error, first one was pypdf, so i installed pypdf, then doc2text giving error when trying to upload docx file.. isn't there some way to automate all these dependencies? Appreciate any help, thanks.

13 comments

WWhiteFang_Jr

Llama-Index does not auto install all the file required library on its own. Otherwise there would be lots of dependency while installing llama-index itslef.

You'll have to download each of these requirements on your own. lol 😅

MMez

is there a list of all the dependencies somewhere?

MMez

i swear on my dev server it was auto installing them as they were needed on the fly...

WWhiteFang_Jr

There is no list as far as I know.

MMez

so if i want to support 20 different file types, i need to dig through the code and figure out the dependencies? that doesn't sound right, or is that just the way it is? on my dev server i could see it installing live the dependencies it needed whenever i first uploaded a file of a type.. so damn weird..

MMez

on my dev server i never had to install any extra dependencies and every file type worked.. freaking bizarre... it was a fresh server setup too... so pypdf and doc2text were definitely not already installed..

WWhiteFang_Jr

I think so yes. Right @Logan M ?

LLogan M

Yea the only time things are autoinstalled is if you use download_loader

I usually just install things as I hit them 🤷‍♂️

@WhiteFang_Jr is right, managing the dependencies of 100+ loaders isn't really feasible

MMez

@Logan M so you're saying that download_loader("SimpleDirectoryReader") would automatically install pip packages when the code is trying to execute?

MMez

because that's what was happening on my dev server but i think I've setup the prod server a bit different and it's just throwing exceptions every time a new file type is being uploaded

MMez

Looks like you are right, thought I was losing my mind:

download.py

if os.path.exists(requirements_path):
try:
requirements = pkg_resources.parse_requirements(
Path(requirements_path).open()
)
pkg_resources.require([str(r) for r in requirements])
except DistributionNotFound:
subprocess.check_call(
[sys.executable, "-m", "pip", "install", "-r", requirements_path]
)

LLogan M

👍👍

MMez

I blew away the venv, created a fresh with just llama-index and as soon as i uploaded a .docx it automatically installed docx2txt... wasted a day on this :/

appreciate the tip man, you sent me in the right direction, much love

Add a reply

Find answers from the community

losing my hair over this I ve installed