I don't treat memory like RAG. That's the key. I only track decisions, actions, and outcomes.
The point about memory is sometimes you remember great detail, sometimes you only remember that the memory exists, so having a good tool loop to attempt to recall and try permutations is good.