You can add a tag to each memory indicating who wrote it, and then at read time filter the memories instead of putting them in the prompt.
I am the founder of Substrate, a project that lets you inspect what is in the memory and select what each agent sees.
Do you separate memory per user with your harness?