The best solution I've found is: each compiled library doesn't just have a single API, but rather has multiple.
libfoo-1.2.3 might implement the following APIs:
* libfoo-1.0
* libfoo-1.1
* libfoo-1.2
* libfoo-1.2.1 (because people don't actually use the "no API additions in patch releases" rule)
* libfoo-1.2.2
* libfoo-1.2.3
* libfoo-experimental-1_2_3 (no semver in the version here)
and if you enable experimental exports for the library, you gain a package-manager-level dependency on libfoo-experimental-1_2_3. Then later you can declare "libfoo-1.3 includes libfoo-experimental-1_2_3", so old programs can use newer libraries without change.
That said, mutable metadata is still an important part of the ecosystem. Blacklisting is essential (since the symbol-list isn't actually the whole API and it is impossible to automatically detect violations), but easily abused for "that's insecure" and "I don't support that" purposes so there should be some friction.
Note also that ABI (and sometimes API) is fundamentally platform-specific. And there's enough complexity in the world that the tooling shouldn't think in terms of semver, but rather arbitrary compatibility DAGs (but with sparseness when possible). Consider the GCC 4.7.0/4.7.1 std::list breakage that was reverted in 4.7.2, breaking software that was compiled in the mean time (the only reason this wasn't catastrophic is because nobody actually uses std::list) - that intermediate software needs to know it can't use new binaries.