The temporal lobe got its name because of its proximity to the temples, so it would be incredibly serendipitous/prophetic if it turns out to actually encode temporal information.
Were temples not themselves often used to encode [spatio-]temporal information (about the time of the year, our location within the solar system and cosmos, etc) within their construction, now that you mention it?