Size really is issue when dealing with mobile devices. Even from disk cache each KB increases startuptime by about 1ms (and jquery 2 is ~32kb minified, for < 2.0 lot more). Over cellular connection you can nicely multiply each KB by 100ms. This is even worse when you minify and concat your JS library dependencies, as most likely your cache headers expire way too fast for them. If you have bothered to set them at all.

Also, last time I checked jquery uses "querySelectorAll" for that fancy $("selector") syntax, which is slow as hell with every possible browser compared to "getElementBy*". This might not be issue with desktop machines, but you will probably lose most of your mobile users because of that. Jquery does also exposes very dangearous things from the API. For example: most of the time use of css modifying from JS just implies that your UI logic is fundamentally flawed (also, incredibly slow as poking CSS from JS will force full relayout which will make your mobile users throw their devices to wall or leave your site).

Jquery lets people cut corners which will give short term benefits, true. But in long term you are just killing your user experience. And like others said, people mostly use like 1% of the API, which has as easy native browser support.