Guessing it's a bunch of trial and error right now - these systems are pretty hard to predict in general and I'm not aware of a whole lot of research into closed ecological systems. Which is a shame really.

A self-sustaining system like we'd want should come to its own dynamic equilibrium, given a decent enough starting state, which does make the trial and error slightly more feasible, and of course they have to have some idea of the inputs and outputs of the various organisms.