Agriculture was (is?) a disaster for our health.
Edit: I noticed that you spoke about increased energy intake and brain size, not just health.
Agriculture paved the road the specialization, and hence the progress, diversity, safety and richness we see in our lives today. We can argue over whether these are virtues or not, but we can't turn our heads from the Darwinian view of life. Living as a nomad, with everybody toiling just to feed the mouths doesn't scale. And, Natural Selection says any group that is able to specialize even a little bit gains strategic advantages over non-specialized groups. No wonder hunters and gatherers were driven to extinction.
Is Jared kidding us when he says that social echelons are bad? I guess, he thinks our hunter-gatherer ancestors treated everybody equally and with respect. I guess 30 seconds in front of Discovery Channel would tell him how apes live - alpha males "ruling" over females, and killing other males over food and right to reproduce. I'm quite amazed at his ability to scorn at social echelons and other malices brought upon us by agriculture while giving a silent nod to killing for food in the same breath!
Though that paper makes some difficult-to-verify claims.
It might be less a result of the diet's intrinsic problems and more a result of evolutionary pressures favoring dumber people, because work is much easier, so investing in intellect wastes energy that could be spent on other things.
Human brains have shrunk since the establishment of agriculture. Brains are smaller now than during even the middle ages.