If you use Hermes, one tip: `session_search` allows searching through all past sessions via full text search. So you don't need to load the notes in every turn.
Another hard part is that your vault of notes and the agent's memory are separate. They can drift apart. We, at Substrate, have solved this with our memory architecture. With it, all of your notes are considered as one memory, and we can always attribute a source to each fact.
What notes does your agent use in a given session? https://trysubstrate.co/l/a2hn