I use both OS X and Ubuntu. In comparison, Ubuntu is terribly slow, even on basic things such as dragging and resizing windows, let alone graphically-intense applications. This is both with ATI's proprietary drivers and the opensource radeon driver. Things are a bit better if I don't use a compositing window manager.
I am not sure who is to blame, X or the drivers. But most users won't care. They'll see that GNU/Linux has a bad desktop experience, and revert to whatever they were using.