For example, I like to compare snapshots to a rolling greyscale 3s average image and extract contours around large diffs. This still yields lots of noise (moving shadows are the worst!), but I don't need realtime answers and have a relatively fixed domain. This let me add models like CLIP to search out specific known false positives (and known good positives) to improve detection quality.
A moving camera, obviously has no fixed frame of reference to detect motion. As such, your heuristics need to change, and they'll get considerably more complex (and computationally expensive). Honestly, I wouldn't even know where to start, and likely wouldn't even be based on OpenCV.
If anyone has a starting point, please do share.