hi guys, quick question - is it possible to save documents to Disk? I couldn't find any reference to that in the doc.
I have a pipeline that needs to read documents from disk and index them. The fetching and the indexing happens on 2 different microservices. I want the first service to first store the documents to disk (S3/GCS) and then have the second service reads those documents and embed them

Hi everyone! I have a general question about RAG and Data Privacy. I'm using llama-index to build an internal Q&A chatbot, which is fed by multiple data sources (Slack, Confluence, Jira, Google Docs).

Now, when a user talks to the bot, I want to fetch documents which this user is allowed to see. For example, if a user is allowed to see document X but not document Y, I want the semantic search to exclude document Y.

What's the best way of doing that? Are there any best practices around this issue? I couldn't find much information online, and specifically about llama-index + privacy.

Hi everyone! I have a general question about RAG and Data Privacy. I'm using llama-index to build a Q&A chatbot, which is fed by multiple data sources (Slack, Confluence, Jira, Google Docs).

Now, when a user talks to the bot, I want to fetch documents which this user is allowed to see. For example, if a user is allowed to see document X but not document Y, I want the semantic search to exclude document Y.

What's the best way of doing that? Are there any best practices around this issue? I couldn't find much information online, and specifically about llama-index + privacy.

Hey everyone, I have a general question:
I'd like to finetune a model on several data sources of data. I'm using the llama-index loaders to load data from multiple data sources. What is the best way to save this data and fine tune an LLM on it? Should I build a vector DB without embeddings and once training is started - fetch the data from the DB? Or should I just save the data to GCS/S3 and load it from there? If the second option is the "correct" one, is there a built-in way to do that with llama-index?

Hi everyone, what's the easiest way to finetune an LLM model (not embedding) on my VectorDB?
I already have a RAG pipeline inplace and I'd like to fine-tune open source models to enhance the overall usefullness.
I saw the llama-index finetuning doc, specifically the part with Llama2 + Gradient.AI but I was wondering if there is a more transparent way without paid tools:
https://docs.llamaindex.ai/en/stable/optimizing/fine-tuning/fine-tuning.html#fine-tuning-llama-2-for-better-text-to-sql

Hey everyone, I'm having issues with OpenAIEmbeddings in the new 0.10.6. It seems to crash due to an assignment issue of the callback_manager. I opened an issue with more information here:
https://github.com/run-llama/llama_index/issues/10977

Did anyone else experience that?

Hi guys, I have a question about GithubRepositoryReader. I have encountered a rate limit by Github. What is the best way of using this loader? I tried to provide it the default branch name, but I suspect it is "too" expensive because it tries to collect all commits. Should I use a commit sha in order to overcome the rate limiting?

Hey everyone. Does anyone knows a VectorDB which supports clustering algorithms like K-Means, DBSCAN, etc?

Hi everyone! Great to be here 🙂
I have a question - I'd like to use LlamaIndex in our project in order to fetch data from multiple sources.
Should I use the Python framework or the Node framework? I'm personally knowledgeable with both languages. I care a lot about maturity and it seems the Python one is the more "mature" project. Would you recommend using it instead of the Node.js one?