Find answers from the community

Updated 3 months ago

I'm using 0.10.5 in an AWS Lambda for

I'm using 0.10.5 in an AWS Lambda for handling queries from a frontend client. On this line, according to the logs, it tries to download NLTK ... why?? I'm also getting a lambda error, "read only file system" so I want to avoid downloading nltk.

Plain Text
from llama_index.embeddings.openai import OpenAIEmbedding

logs:
[ERROR] OSError: [Errno 30] Read-only file system: '/var/task/llama_index/core/_static/nltk_cache/corpora' Traceback (most recent call last): File "/var/lang/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load ...
L
r
8 comments
Hmmm nltk is supposed to be bundled with the pypi wheel
maybe this isn't working at the moment... will take at a look at the publishing pipeline
I added nltk to the requirements.txt for my container, and it appears not to force download when calling from llama_index.embeddings.openai import OpenAIEmbedding.
here's my requirements.txt

Plain Text
openai >= 1.2.3
llama-index-llms-openai == 0.1.1
llama-index-embeddings-openai
llama-index-core == 0.10.5
llama-index-vector-stores-opensearch == 0.1.2
opensearch-py
nltk

while I have you here. What is the difference between llama-index and llama-index-core in this requirements.txt file??
@Logan M I have to revert back to an earlier version of llama index. I don't need nltk stopwords to use openai embeddings but it is downloading it anyway (and causing problems in aws lambda)
llama-index includes things needed to get started quickly. It has similar requirements to the old llama-index package.

llama-index-core is more barebones, with no included integrations
I know the issue with nltk packaging now, will make a fix a for this and publish a new version of core πŸ™‚
thank you @Logan M !
Add a reply
Sign up and join the conversation on Discord