In the old days, some software would actually check the terminal's baud rate (which you can see with 'stty') and behave differently on slow links. For example, vi would restrict itself to fewer lines of the screen so that drawing a new page didn't take as long.

Although I don't know if much software does it, that should still be possible to do it, so these animations could theoretically be auto-disabled on slow links. (And you could manually override the baud rate in your mosh terminal if necessary since it doesn't really have any other effect in a virtual terminal, not like when using an actual serial port, I mean.)

It shouldn't matter, should it? Mosh keeps the state and only sends the differences. You would miss most of the animations, and there would be the overhead of sending the colouring escape codes (a few percent extra), but other than that, it wouldn't be any slower than an unanimated version that I can see.