lol yea it just hallucinated using the date tool
I would modify the description to the obsidian tool to mention that it can be accessed by date
The correct place to do that is in the description of the tool?
Is there a "best practices" documentation?
not really -- a lot of this is just prompting 😅
It also looks like you are using a react agent? Not sure which LLM you are using, but open-source LLMs are generally bad at being agents these days
If you are using openai, I highly recommend the OpenAIAgent
LLMs, currently llama-2-13b-chat
might be the worst agent lol
What would you recommend? For agent and local llm? Checking zephyr right now...
zephyr was probably the best I've tried. Still slightly unreliable though, agentic behaviour seems tough for these LLMs
For example, some results from a recent paper benchmarking on a ton of datasets
Ok, so you are saying it's basically an issue with the llm not understanding what's going on and not a code issue.
pretty much. Some prompt engineering may help of course, but feels like a general issue with open-source LLMs at the moment.
They can power little demos, but beyond that things get tricky with them
That's sad. I don't want to put my obsidian vault into openai or something.
Things will improve someday 🙂 They work fine for general Q/A and RAG though.
For example, in your case, I would just include todays date as part of the prompt
Ok hm lol. Now with llama-2-13b it "somehow" works, including some hallucination.
It shows the correct note of today, but thinks today is a saturday.
That "tool description" is used as part of the prompt would be my guess?
If I ask for yesterday's note the reasoning itself works well, it uses GetDate and then reasons that yesterday was the 18th. But then it crashes in the react/step.py in some reasoning step "could not parse output"
yea, the LLM is shown the chat history, list of tool names + descriptions, and some instructions for the ReAct format
Hm ok, anything you would recommend to read or yt to get a better understanding how things work?
Thanks a lot. For the help and the links.
