Take a look at the caniuse tables for common functionality in JQuery:
http://caniuse.com/#search=classList
http://caniuse.com/#search=querySelector
http://caniuse.com/#feat=css-transitions
http://caniuse.com/#feat=getcomputedstyle
http://kangax.github.io/es5-compat-table/#Array.prototype.fo...
There isn't a one-size-fits-all answer for this, but you should have some idea of what your traffic breakdown by browser is, and ideally what your cost in conversion rate is for each additional ms of latency (this varies by industry). I don't remember offhand what the cost per byte in latency is, but IIRC we measured something like 1ms per 1K bytes at Google (this would work out to a 1 MBps = 10Mbps connection, which seems around right for typical cable/residential fiber households these days).
As far as latency is concerned, I hear you and that makes sense, but the things I'm talking about are not very big, so the latency gain isn't going to matter much for our purposes.
Yeah, we definitely look at the analytics. But even if it the old browser users are sub 1 percentage point, I see the number and think, "I could take this 5 line polyfill out, but then these 1000 uniques wouldn't work."
And I just don't have the heart...
I thought that the issue in this thread is JQuery, which is a layer on top of the modern browser API (probably because the modern browser API didn't exist when it was created, and is in a large part a native implementation of JQuery functionality), and so incurs the cost of all its legacy compatibility support even if you don't need it.
For anyone else, just use jQuery.