Find answers from the community

Updated 4 months ago

File loaders

At a glance
The post asks about the functionality of file loaders, specifically whether they return results to search queries and chunk data into "pages". The comments suggest that file loaders simply return the data from files, which can then be indexed using a tool like Llama Index, allowing users to ask questions of the index. Community members discuss the need for file loaders to handle hierarchical data structures like org-mode or JSON, to avoid "orphaned" child nodes and maintain context during searches. They propose ideas like building a smarter parser to preserve hierarchical information, and adding time-based weights to the data chunks. The latest JSON loader is mentioned as potentially addressing some of these needs.
What do these file loaders do any ways? Will they return results to a search querry and chunk data in to "pages" that get returned?
L
y
8 comments
The file loaders just return the data from files, so that you can index it with llama index.

Then you can ask questions to your index πŸ‘
thanks @Logan M
should the file loader do any pre-processing than?
org-mode, which I want to parse, is very hierarchical. Like json.
I don't want children to be orphaned for search. If I search for "Peters Birthday" it has to find the birthday node under peter.
I finally managed to load my file generically πŸ™‚
but I feel the lack of context right away...
Yea that's understandable! It's a tricky thing.

You could build a smarter parser that tries to maintain the hierarchical information.

Like parse each child node into a string that contains the parents

people -> Peter -> birthday -> April 7, 2023

Just a rough idea though, but something like that πŸ€”
yea thats what i thought (and hoped the json parser does and i just need to fork it...)
I'd also love to add weights to the chunks: older = less important
I think the latest json loader actually does this, was just reading more about it yesterday πŸ˜…
Time filters/weights hopefully coming soonπŸ™
Add a reply
Sign up and join the conversation on Discord