When going frame-by-frame you tend to go back and forth. You can preload and cache, but that leans a lot more on your player and hardware. Yes, you do get every frame but some frames may be very "muddy" because of compression. I'm not saying it's impossible, it's just not an optimized use-case that's trivial with other codecs.

Seeking is another example. It's a hard requirement for video editing, and desirable for users but often compromised for more preferable things like bandwidth and file size. For example, if the data-rate is variable, you don't really know where 3minutes 20seconds into the file is without hunting around (which is trivial if the data-rate is constant). MP3 supports a 1-100% lookup table at the front of the file, but this isn't very granular for long files. Even if you can find the spot in the file, you'll probably snap to an i-frame unless you decode the whole segment.

I'm sure these things will get better as codecs mature and constraints on size and processing power ease up.