I've found I don't really want my agents to remember very much.
The projects need to remember things, not the agents.
When they're a blank slate coming in, I get more reliable outputs.
I just have one orchestrator/secretary/router with long-term memory, but I don't switch its harness.