Except that jQuery Core is a completely different thing. You know if you're using jQuery UI Menu but not jQuery UI Dialog so you can decide to include them or not with a custom web-based builder just by clicking boxes.
The jQuery Core features used by all the plugins you collected to build your site are not immediately obvious. Did they use .replaceWith() for example, is it safe to leave that out? What about .load(), did they use that API? Oh, wait, there are two different things that .load() does, one is AJAX-y and one is event-y, if they did use it which one did they mean?
By having the full library there you don't need to spend lots of time tracing down these dependencies. And in JavaScript these are all dynamic, you can never really be sure you got the dependencies right until you find out you didn't.
That's why I strongly advocate that the average programmer should just use the whole standard jQuery core library unless you are in some spot where saving 10-20KB gzipped is so important that you're willing to chase down those dependencies yourself.