The post asks about the functionality of file loaders, specifically whether they return results to search queries and chunk data into "pages". The comments suggest that file loaders simply return the data from files, which can then be indexed using a tool like Llama Index, allowing users to ask questions of the index. Community members discuss the need for file loaders to handle hierarchical data structures like org-mode or JSON, to avoid "orphaned" child nodes and maintain context during searches. They propose ideas like building a smarter parser to preserve hierarchical information, and adding time-based weights to the data chunks. The latest JSON loader is mentioned as potentially addressing some of these needs.
thanks @Logan M should the file loader do any pre-processing than? org-mode, which I want to parse, is very hierarchical. Like json. I don't want children to be orphaned for search. If I search for "Peters Birthday" it has to find the birthday node under peter.
