It seemed it was detached DOM elements that were causing most of the problems. We tried profiling with the Chrome dev tools but found it very difficult to pinpoint where to start looking from the thousands of elements generated every-time we repeated our workflow.

In the end we looked at the bottom line memory consumption and experimented until we saw reductions. We found using things like ng-show instead of ui-if, essentially preloading the partials and switching between them instead of reloading everytime, saved us enough memory to make the system viable.