Is there a performance problem with losing proprietary GPU drivers?

Keeping them seems one benefit to dual OS (reusing the Chrome kernel). It's nice to be able to watch HD video without hiccups.