But not putting one, using foreign code base might get you in trouble (resource leaking, not de-initializing certain piece of the parent, threads not closing, etc.).

This would've been prevented if destructors were always virtual.

My take is that if the compiler cannot make the decision, but it's left to you, in this case it's better to always have the virtual destructors ON.

The problem is more visual than analytical - you would have to dig through the whole hierarchy in several files to understand whether you need to put a virtual destructor or not. Compiler/Linker/some post-proc tool should be better at that job. It already has all that information.