My PhD (before 10+ years) was in video search and one of the proposed methods for video comparison was using the shot durations of the video. Notice that with shots I refer to cuts in the camera flow, IIRC hollywood movies have a such a shot cut every 4 seconds as an average (for example when two people talk the camera will move from one person to the other).

I remember that I used two techniques for extracting scene cuts:

* Difference in the brightness (Y) histogram of the YUV video between frames; when that difference is more than a threshold there's a scene cut

* Counting the number of Intra macroblocks per frame on an H.264 encoded video; if that number was more than a threshold then there's a scene cut