Hi team hope all are well I have a

tthomoliverz

Hi team - hope all are well. I have a slackbot at my company using llama-index to read notion pages and answer questions. This works well but 1 issue. The info in the pages is dynamic and constantly updated, as we improve our knowledge base.

Anyone can make a change to a notion page but I am required to re-run my script (on Replit) whenever a change is made in order for those changes to be updated in the index.

Is there a way to have the script re-run/ load at regular intervals?

Ty for any help.

7 comments

LLogan M

Could you just have the script on a cron-job ?

tthomoliverz

sorry would you mind explaining what that entails?

LLogan M

So, linux-based systems have this concept called cronjobs

Basically, once cron is running, you can setup scripts to run at regular intervals
https://phoenixnap.com/kb/set-up-cron-job-linux

Debugging these can be a little annyoing, I usually funnel all the script output to a log file somehwhere

LLogan M

The time format can be a little weird, I usually use this website to help me confirm the timing
https://crontab.guru/every-15-minutes

tthomoliverz

thank you - mega kind and will look into.

rrawwerks

@Logan M - what are the best practices for cron reingestion? delete the entire vectorstore and start from scratch? my specific desired use case would be to recursively re-embed a website on a cron schedule. for example, once a day, find and index every page on example.com/* and load into the vectorstore.

LLogan M

With the recent v0.9 update, I would delete/recreate every time, but use a cache in the IngestionPipeline

https://docs.llamaindex.ai/en/stable/module_guides/loading/ingestion_pipeline/root.html#ingestion-pipeline

Add a reply

Find answers from the community

Hi team hope all are well I have a