They are libraries, but they are used so fundamentally, that replacing them in a existing app is a pain, so most of them impose the problems of frameworks too :\
The pain of removing jquery is indeed a problem. So I can see the point from that angle. But the bigger point, about how much less assuming they are just by virtue of being included... that's the distinction I cared most about.
I too am confused by the claim that Dojo and YUI are not frameworks, but Backbone is.
Yeah, all three have Collections and Views.
The fact that Dojo and YUI can be used like a framework (so can jQuery, with jQuery UI, etc) is not the point. The point is that, at the point-of-inclusion, those 3 tools/libraries are far less intrusive to the architecture of your site than including Backbone/Angular/Ember.