>Even if there was a way to gain efficiency through self modification rather than losing it, it would likely hit a plateau of diminishing returns, like most things in the world do.

And maybe you're fine with betting your life on that, but the people who are more-hesitant on this are not crazy.