The not-implemented instruction trapping in the 68K could be easily evolved to cover less frequently used instructions to make room for more optimized implementations of the frequently used ones.
What really killed the 68000 was the move to RISC, in particular Apple's move to PowerPC. That took away any hope of future evolution (they even managed to release the 68060 after that, but that was it) and collapsed the high-end 68K business.