Isn't that exactly what genetic drift is?