The Roman empire had plenty of places to expand to (much of Africa, all of Russia/Asia). Yet they didn't manage that.
This explanation also strips agency from the various peoples who made their own decisions to setup autonomous Roman style rule, throw off Roman rule, or peel off Roman territory and traditions for their own pre-existing polity.
As you point out, a whole lot of the Empire's subjects had a very different take on the matter.
On the other hand, from what I understand, there is some evidence that there was a significant pick up in the rate of technological innovation after the fall of Rome. Not sure how that stacks up against the technologies lost in the early Middle Ages.
The focus on the decline of public works and art is fundamentally an elite view, since they were never coerced to labor for them or die in the fields protecting the empire. For the common man the empire was much worse, but the common man doesn’t write the history books.
Still, I have trouble imagining a decline in economies of scale / specialization that large scale trade brings wouldn't have harmed the quality of life of the common man.
Toxic heavy metal poisoning.