Yeah, you really need something like they do in Mission Impossible 4 where it has a single target whose eye it's tracking, changing the perspective projection accordingly.