J2EE's sins include far more than just EJB. See http://en.wikipedia.org/wiki/Java_Platform,_Enterprise_Editi.... Servlets got just started running with a bad crowd, even though deep down it was a good kid.
Heck, the architects even designed it as a completely generic request/response framework with http as just one possible protocol. When http is your 99.9% use case, that just smacks of over-engineering.
The Servlet spec was finalized more than 15 years ago, when CGI scripts were the mainstream server-side tech and browsers were circa version 3. The domain "google.com" wasn't registered until 3 months later. I think Servlets were reasonably well done for the time. The web was obviously dominant, but FTP was still widely used, and it certainly wasn't clear that newer protocols wouldn't be important as well. Rails didn't turn up until 8 years later.
One could certainly blame the java folks for stopping there, but there wasn't a lot of advantage to be gained from redoing the foundations in a way that forced everybody to rework all of their existing servlet code. And as you say, the community was making solid progress on their own, so it's probably better that Sun didn't interfere.
In my time working with Java (14 yrs now...sigh) for simple Web applications, I've never found the need for the majority of the J2EE stack, instead opting for Servlets and...well, just normal Java code. I've had Java gurus throw proverbial books at me for not using EJB and its ilk, but I just don't like how complex it all gets once the full J2EE stack is included.