So, it's a mixed bag. "Linux" means so many things that it's hard to say "Linux is easy to use" or "Linux is hard to use" and not be right in both cases.
1. My soundcard didn't work with any vanilla kernel (+ modules)
2. I owned a webcam that had a slightly different USB id than the ones supported by the driver that in fact also supported mine.
3. I owned a laptop with a touchscreen which involved having to compile some non-standard modules.
Anyways, things have gotten a lot better since I've "left", but Linux (on the desktop and Android, for that matter) will always be "very good the next release". Which is, of course, utter bull. If you don't have excellent polish as a priority from the start, you'll always be fixing things for the "next" version while simultaneously introducing 10 new half-baked "features".
Even though I still manage a bunch of Linux servers and advocate towards Linux whenever it makes sense.