They were betting their existence on a species they knew already uses MAD as a basis for its' survival not figuring out that it could use MAD to defend themselves.
Their system was interesting, but ultimately not much worse than ours, once you leave the planet. And they have to leave the planet anyway.
Once you have nanomaterials allowing cheap access to space - putting your population in space habitats is cheap and easy. We could do it now, if we had space elevator. The cost of materials and energy is negligible compared to invasion of another star system. Especially with such conveniently storeable population :)
> Tri-Solarians knew far more about Earth than they could reliably determine about most nearby star systems.
There's nothing stopping them from figuring it out. You don't have to invade a system, you can send a stealthy probe. I would assume in their circumstances it would be a wise thing to do if they really don't want to live in space further away from the center of gravity of their star system.
Besides, if the dark forest is so important - why invade Earth at all and play with fire? You can kill everybody on Earth without bothering to show up or sending any warnings. Send a big meteor their way or make a deadly disease that looks like flu, infect everybody, and on set date kill all hosts. Much cheaper, and no risk of ceasing to exist.
What they did in the books was very risky and wasteful for no good reason.