My suggestion would be to keep things simple, pragmatic, and save development time. While git worktrees are not perfect, and they require extra space, it is easier for people to understand and easily locate this kind of structure while also being able to execute commands in a somewhat isolated environment. I would be happy if the app simply automates this for me and creates a new worktree, branch, and agent at that location as soon as I click 'Add agent'.
Then the only issues become merge conflicts between different branches... this is where extra dev time could be allocated, and implement agents that automatically merge branches.