Pretty sure that's how modern video codecs work too. A lot of movies have regions that don't change much between frames (e.g. backgrounds, or some dude's face staring at the camera all dramatically).
What GIF gets you is a lossless video, which is overkill for most applications. Video codecs can achieve a similar perceptual quality at a tiny fraction of the size, which is why services like gyfcat and imgur will often auto-convert gifs to videos for browsers that support it.
Take this 2MB GIF (https://blackmagic.so/assets/sidebar/most-engaging-hours.gif) and put it through an online converter: https://ezgif.com/gif-to-mp4/ezgif-7-68b177ec1944.gif
WebP: 1.64 MB
MP4: 300 kB
AVIF: 274 kB