There's pretty strong indirect evidence for such periods in the fact that life is so much better than it was in say 500 AD. If things only got worse, they'd be pretty terrible by now, starting from that base.