Because X is client-server based, it actually performs much worse than Windows 95. That might be one of the issues he would have with using Linux. It's very high performance as a server platform, but it does a little worse as a consumer OS.
Anyway, I doubt any overhead of the X protocol is visible on a modern machine. With two cores, the client and server can both run at the same time, and server operations are so fast you wouldn't notice anyway. I certainly don't notice any delay, although I do see that X has used 495 minutes of CPU time on my machine (in 9 days). (Compare that to 285 minutes of CPU time that emacs has used, and 137 that Firefox has used.)