I used to use this approach as well for managing many agents and myself. I do think the bloat problem is real as well as the need to handle multiple disjoint workstreams.
My solution was to put tasks and their dependencies in a sqlite db. https://github.com/mhalstrom/maximizepm take a look if you're interested