Earthquakes and other geological events can speed up rotation though, but so far never enough to remove a leap second.