Unwinding changes to the TLB on every mispredict would have a significant overhead and hurt overall performance. Removing valid data you just cached (speculatively or otherwise) is generally a bad idea.

User mode software requires a TLB (unless you want to do a page walk for every single instruction!)

Even if you could remove the TLB entirely from the CPU somehow, the attacker could just use the cache or some other microarchitectural structure.