The industrial organization failure is understandable. It’s all well and good to say “they should have recognized it wasn’t working and stopped” … but even if that’s true at the macro level, it’s very hard on an individual level.
An established physicist working on string theory has every incentive to keep building on that line of research. They have no competitive advantage switching to a new line. The specifics about how much expertise translate how far is beyond me, but it seems that nearby better ideas would have been found given the resources poured into this.
This is exactly The Innovator’s Dilemma. It’s even worse in academia than industry because the whole sector is so insular. I look forward to the reinvention of higher education and theoretical research…