I used to feel like that too, but Backbone is really lightweight and unopinionated. Check out this step-by-step guide on how to convert a jQuery app to Backbone: https://github.com/kjbekkelund/writings/blob/master/publishe...
For situations where I actually do want some kind of client-to-server data-binding, I generally prefer it to provide two-way view<->model binding so I've been leaning towards Angular (as heavy as it is) and Knockout.