They are libraries, but they are used so fundamentally, that replacing them in a existing app is a pain, so most of them impose the problems of frameworks too :\
The pain of removing jquery is indeed a problem. So I can see the point from that angle. But the bigger point, about how much less assuming they are just by virtue of being included... that's the distinction I cared most about.