There is research going on right now to build native camera systems that emit data (at the hardware level) that is not dense but rather sparse; information is transmitted as "blocks of pixels that changed between frame two and three" rather than "here is the state of all pixels at frame two, and here is the state of all pixels at frame three". In essence, doing a small amount of delta compression within the hardware itself.
This seems like a really natural fit for an application such as this one; and could potentially cut the amount of data needed to be paid attention to down by orders of magnitude.
Many of these ideas are similar to what is used to compress video down in the first place; it would be interesting to see a combination of video compression technology and "region of interest" ML pruning, where the ML models are applied only to interesting regions, as identified by the compression codec. The codec already has done the analysis to figure out which portions of the video have changed, which portions of the video are simply shifted versions of information from the last frame, etc.... You could significantly speed up ML analysis by making use of that kind of information.