Let me add a data point in the other direction. The first ~six months of my kid's life was by far the worst time for me.
Yes, there were other challenges as he got older. But the sleep deprivation of that first period was awful in a way that is difficult to put into words.