Automating the Process of Re-running Ingestion Pipelines

I think this generalizes to anytime you want to rerun ANY code/program. There's many many ways to do that 😅

haha i see. i mean my idea is for a small enterprise use-case. Suppose we have data sources from google drive, airtable, affinity, dropbox etc.

And I want to host a private chat RAG app. Ideally every 1 hour or so it fetches the latest data from these sources.

I prob want to host in on the cloud eventually. Any pointers? @Logan M

im looking at pathway already. and i got it to work with google drive. except the responses are so poor... starting to think a vector db approach would be needed

Or i just use a job scheduler??

Cron tab with docker exec rag poetry run generate