I just discovered that Zoom teleconferencing software has the ability to detect the background and replace it with an arbitrary video in real time. I have no idea how they do it, but it's as impressive as heck.
- a background scene is usually static. if a pixel matches its historical average (i e. not changing) then it's probably background.
- if a pixel color matches its neighbors, it's probably the same as them. i noticed my black shirt tripped it up on occasion, but was all-or-nothing.
- people are blobs, not diffuse, so try to segment large regions.