Driving e-ink displays faster usually wears them out more quickly.
Another issue I heard, but I don't know enough about the physics of e-ink to know if this is correct, is that if you are at a gray level, the white and black particles are mixed and might exchange charge, thereby causing some particles to not move correctly anymore.