There is a lot of previous art on this sort of thing, like the ffmpeg spp filter: https://ffmpeg.org/ffmpeg-filters.html#spp-1

It's unclear to me if this method will perform better than those methods.