I recently heard that the same brain regions are involved in processing temporal and spatial distances.
If you happen to recall the outlets you read this at, I would be interested in learning more. Hopefully the article would have links to papers as well... perhaps including fMRI scans? Truly fascinating stuff.
Think of time spans as "percentage of life lived so far", and a lot of that particular time stretching suddenly makes sense.