Just giving up speculation entirely is too much of a performance penalty. So one possible solution is: when you know it was a misspeculation, processor can just kick cache lines out of cache if the speculation caused the cache line to be loaded (probably you only need to do this at the last level cache closest to the memory). More states to track, but not impossible.