YUI was more extensible, followed (mostly) good JavaScript patterns (making heavy use of the module pattern), had relatively good cross-platform support and an easy to understand interface.
To top it off, the documentation was top notch.
To top it off, the documentation was top notch.
Serious? That's what drove me away. It might have been top notch academically but for real life use I found it painful to get answers and simple code problems fixed.
The top hit on Google for "make an ajax request with (YUI|jquery)" the documentation page is the first hit.
YUI's page is here, the line of code to make the request is well below the fold. http://developer.yahoo.com/yui/connection/
JQuery's page has it listed immediately. In fact, 2 variants above the fold. http://api.jquery.com/jQuery.ajax/
I'm not saying one is right and another is wrong. I'm just saying jQuery's gets more adoption. It's easier and people tend to like that.
I don't, for example, understand why I have to scroll past the YUI dependency configurator to get to the syntax I'm looking for.
I just know that when I did try it (2009) I felt the same as I do today.