Find answers from the community

Updated last month

Weird Behavior with SimpleDirectoryReader and Load_data()

At a glance

Hello everyone! Encountered a weird behavior (prob a bug) from SimpleDirectoryReader and load_data(). Namely, when I try to get metadata with multiple workers it bugs out.
'''
reader = SimpleDirectoryReader(dir=dir, file_metadata=get_meta)
documents=reader.load_data(num_workers=4)
'''

results in
can't get attribute 'get_meta' on <module 'main'>

while if I set num_workers = 1 it works fine

Anyone had the same issue?

12 comments

WWhiteFang_Jr

hey your get_meta is a callable func right?

if it is a simple dict I would do it this way:
get_meta= lambda extra_info:{"key":"val"}

iiDyn

yes, it is just def get_meta(path): return {"foo": "bar", "path": path}