I think the Romans were right to believe that, and so are we today.
But the ride wasn't sa clean, steady slope up. For example there's a reason the dark ages are called dark. Most people of the time (except the few educated) didn't know there was something better in another time and another place. They probably thought that's as good as anyone's ever had.
Now we live in the first period in history where knowledge of history is accessible to almost any person. So as a regular guy you can have a good sense of where to place these times on the scale of civilization.