Romero is a strange figure. Why wasn't he able to really make it after id? From Daikatana to Empire of Sin, everything was a flop, more or less. Who is John Romero?
In some ways Romero actually saved ID software and made everyone a ton of money by working out licensing deals for their game engines. Carmack wouldn't be able to continue to work on his tech and hire Abrash (who optimized Carmack's code) without the money coming out of these deals so he unfairly complained that Romero wasn't pulling his weight for the current project and treated him like crap until he got fed up and left.
I don't know enough about the specifics of Romero to say which it is, possibly more than one, but it doesn't seem to be unique to him, I see this kind of pattern in pop culture all the time.
They needed talent to pull it off, but also lots of stars aligned for a bold experiment to work out, and true experiments don't always work out. Conversely, for each success there were many talented folks who tried good ideas and didn't see them work out. The more surprising the first success was, the more foolish you'd be to expect a repeat, it seems like.
If that's the basic problem, you can't "fix" it in the sense that you can't make someone keep churning out revolutionary new things. But you might get better outcomes and fewer embarrassing failures the more you think of individual talent as one factor. Folks still have to iterate, get feedback, work with good people, sometimes put a lot of work into an idea then see it they have to rework it, or even release some quick experiments that don't do that well. You don't maximize your odds of great results by sending someone off to work in isolation as long as they want.
One thing I see hints of here and there, but don't really have a great picture of, is the inner workings of organizations that produce new/weird but successful things more than just once. It interacts with business, luck, and changes in the world outside of course, but internally--how do they balance single vision vs. input from many folks? Do they build a lot they throw away? Do they go outside the team a lot for feedback (in Romero's world "playtesting")? Of course there's no one Right Way to do things, especially across domains, but it could be fascinating to see a bit more how various places work.
First of all, most game designers don't have as much success as he had after Id. Second, how are you going to follow up games as successful as Doom.
Also don't forget Wolfenstein, Hexen..
(Full disclosure: I met him once and he was nice.)
He does seem cool as an older, chill guy now that he doesn't have to live with the pressures of being a hit parade.
Other commenters give more substantial answers, but I can vouch for the book as having a lot of history, clearly being written by Romero himself, and having a lot of self-reflection on the questions you're asking.
Here's some additional HN discussion: https://news.ycombinator.com/item?id=37649594
One thing that kept both grounded was the brutal timing of their releases. Doom had 6 months or so, and if it failed they knew they had to go back to boring jobs.
Apart from that, Masters of doom paints doom-id culture like the 1970's rock n roll band stories. A group comes together, works as well oiled team for a while and produces genius results. But the success and differences in personal growth trajectories inevitably pull the group apart. There is typically a painfull divorse phase between the main egos. The reality was that this kind of synergy never lasts more than a few years in any case.
Horribly paraphrased:
Carmack was able to pull off impressive tech without impressive level designers was the gist of it.
Romero is talented at the design side but at the time that didnt translate into better games / companies like the fundamental tech did.
I’m not sure there really is any surprise here. We all know that being a success doesn’t imply generalizable skill, whether it’s a one-hit-wonder song, game, app, or business.
Who knows.. Even Minecraft was catching lightning in a bottle. It was there just at the right moment at the right time. The odd choice of Java was just the right thing needed to make it take off. First since you could play it in the browser(remember applets?), later when easy decompilation and modding arrived.
They had equal shares and were all very young, the only option for them at the time appeared to be parting ways. Carmack has voiced in multiple more recent interviews that in hindsight there were probably better options to make things work out, but they didn't have the maturity and experience at the time to pursue them.
Given this context I don't think it's surprising that Romero would crash and burn on his own after being pushed out of id. The self-discipline and maturity side was only more critical to success without the id team's support, not less.
When I talked to fellow gamers about the awesome stuff Carmack does, they were all like "he's just a engineer, Romero is the real genius!"
But somehow I didn't see anything impressive from Romero, while Carmack was doing cool stuff all the time.
Sure, it wasn't artsy like Romero's work, but it was still impressive.
Many other level designers rapidly surpassed Romero.
Carmack was doing a great job until he started building rockets.