I was trying to crack exactly the same problem a decade ago - I was very frustrated by "classical" perspective distortions (such as looking up and down at a very tall pole, you notice in 3D projection the projected width at a given height changes depending on the pitch angle, which is counter-intuitive as your eye doesn't do that).
I searched for answers, read Denis Zorin's disertation from Caltech, various different camera models etc.
After one talk with a neurosurgeon he mentioned that those "perspective distortions" are perceived often by patients after a brain surgery and as the brain heals the normal undistorted picture begins to emerge.
I am still curious if there is a more realistic mathematic formula that would allow us to have exactly the same perspective projection in 3D graphics as we have with our eyes. Anyone knows any recent research on this subject?