> and maybe more surprisingly, add/removing virtual functions

I don't have the spec handy but fairly certain that v-table implementation is compiler specific and while it may work it isn't guaranteed. However if you declare the function symbols as dynamic then you can leverage the linker to dynamically resolve the right symbols with the matching opaque data and achieve binary compatibility(assuming you use the same compiler or a compatible compiler ABI and all other caveats around C++ binary compatibility).