Hi, I am trying to use both llama-index

Hi Bordinha. Would you care to share error logs? Most of us won't know without running your code.

sure thing!

from llama_index import VectorStoreIndex, download_loader
import s3fs

def lambda_handler(event, context):
#key = event['Records'][0]['s3']['object']['key']
#folder_number = key.split('/')[0].replace('document','')
folder_number = '1'
input_folder = f'documents{folder_number}/'

S3Reader = download_loader("S3Reader",custom_path='/tmp',refresh_cache=True)
documents = S3Reader( bucket='fedsp-openai-train', path=input_folder)

The error: ContextualVersionConflict: (botocore 1.33.13 (/var/lang/lib/python3.11/site-packages), Requirement.parse('botocore<1.32.0,>=1.31.83'), {'boto3'})

Yeah. Your botocore 1.33.13 is too new. You can manually install a compatiable version.

Here's a related discussion on StackOverflow: https://stackoverflow.com/questions/62537146/contextualversionconflict-when-using-s3-boto3-and-snowflake-connector

So you recomend me to downgrade boto3?

it would be a bit hard, since i am trying to run it on a aws lambda function (which already have boto3 1.31.83 pre-installed

Hmm. That's where my knowledge ends. Sorry, I've never used AWS Lambda before. This sounds like more like a AWS Lambda question than a LlamaIndex question, though -- I'm sure many more AWS developers have bumped into the dependency issue than there are LlamaIndex users bumping into them. Maybe try your luck searching on AWS Lambda dev forums?

You can abstract the aws lambda part thinking of a server, prebuilt, installed with a stable version of boto3 (in this case 1.31.83)

Then it is possible to reproduce my issue in any linux scenario

I see. Can you try using images other than FROM public.ecr.aws/lambda/python:3.11?

Also, even though it's a pre-built image, you can still downgrade a Python package, right?

Your alternatives would be to either 1) build an image from scratch or 2) modify LlamaIndex to be compatible with botocore 1.33.13, AFAIK. I think downgrading one package is easier than both.

this is a excelent idea! Using another image from start

but yet, it would be great if the dependencies match

do you know how to downgrade the package in a docker image?

I usually just build upon the original image with ‘FROM the/old:image’ and then add a layer of ‘RUN pip install package=verison.i.want’. Sorry, typing from my phone.