Whenever I had a chance to run on nvidia binary drivers, the only things I occasionally missed were some new features, or having to wait a bit longer to update the kernel. Stability was better, drivers more performant, and I don't remember daily fighting with memory leaks.
And X.Org's driver architecture could be replaced completely (in fact, it could be made to run on the same stack as Wayland) - it wouldn't be the first compositing Xserver around, and could use methods that would deal with noticeable to many lag involved in compositor-based UI.