As if it was just that easy to remove all part of third party code ... For example removing a physic engine from a game is pretty much "impossible".
Of course, every physics engine behaves a little differently from the others, so, if you game is physics centered (pinball, racing), switching implementation might result in a slightly different game - but having the core of your game depends on the implementation details of some 3rd-party library might not be a good idea anyway.
In general, the more you sprinkle your code with dependencies to 3rd party libraries, the less control you have over the resulting product.
Your point still stands, though, for some very low-level "utility" libraries like boost or the STL, or any standard library replacement , and more generally, libraries holding "vocabulary" types.