Hey people

Hey people,
I have this

Plain Text

s3 = s3fs.S3FileSystem(
    key=AWS_KEY,
    secret=AWS_SECRET,
    endpoint_url=S3_BUCKET_URL
)

def add_new_doc(name, filename, index_set):
    with open(filename, "r") as f:
        doc = f.read()
        storage_context = StorageContext.from_defaults(fs=s3)
        index_set[name] = VectorStoreIndex.from_documents(
            [Document(text=doc)],
            service_context=service_context,
            storage_context=storage_context,
        )
        storage_context.persist(persist_dir=f'bucket-name/{name}')
    return index_set

however the index for the specific file is still being saved into the local file, and doesnt go into s3, there are no errors

3 comments

LLogan M

@JF I think you need to pass in fs when doing persist as well, the fs object doesn't seem to be saved to the storage context itself 👀 (just looking at the code)

JJF

yeh i saw that, i forgot to send the message where i was "ranting" about why the storage context takes fs but it isnt used anywhere
thanks for looking into it and letting me know i wasnt being dumb and missing sth baout the fs in storage context

LLogan M

yea. in the form_defaults, the fs object is just used for loading 🙂 Glad you got it figured out!

Add a reply

Find answers from the community

Hey people