Try scrubbing backwards. H264 seeking only works nice if you're fast-forwarding the video. Actually, that is kind of magical.