Recent tests with link time optimization give me the impression that code which is not actually called is not included (or even resolved). If you can convince the compiler that the destructor is never called, it might not include it.
Sadly, small oversights (e.g., exception handling for undefined virtual methods) can cause disproportionate bloat.