I'm facing a weird issue. I'm using Astra DataStax to upload my embeddings. The issue arises when I load files from a local directory containing 6 PDF files; however, when I check documents.length, I'm getting 485. Why is that so? How to solve this ?
const dataPath = path.resolve('../documents');
const reader = new SimpleDirectoryReader();
const documents = await reader.loadData({ directoryPath: dataPath });