Find answers from the community

Updated 4 months ago

File loaders

At a glance
What do these file loaders do any ways? Will they return results to a search querry and chunk data in to "pages" that get returned?
L
y
8 comments
The file loaders just return the data from files, so that you can index it with llama index.

Then you can ask questions to your index πŸ‘
thanks @Logan M
should the file loader do any pre-processing than?
org-mode, which I want to parse, is very hierarchical. Like json.
I don't want children to be orphaned for search. If I search for "Peters Birthday" it has to find the birthday node under peter.
I finally managed to load my file generically πŸ™‚
but I feel the lack of context right away...
Yea that's understandable! It's a tricky thing.

You could build a smarter parser that tries to maintain the hierarchical information.

Like parse each child node into a string that contains the parents

people -> Peter -> birthday -> April 7, 2023

Just a rough idea though, but something like that πŸ€”
yea thats what i thought (and hoped the json parser does and i just need to fork it...)
I'd also love to add weights to the chunks: older = less important
I think the latest json loader actually does this, was just reading more about it yesterday πŸ˜…
Time filters/weights hopefully coming soonπŸ™
Add a reply
Sign up and join the conversation on Discord