"what parts of flash are not open?"

The source code of the Adobe Flash plugin, if nothing else.

Incidentally, that does bring to mind one possible Hail Mary that could work: Open source the Flash player(s). Whether the open source community could fix performance issues on Linux and OSX depends on a lot of things, but they could certainly fix some things that would make it less aggravating.

(Since it's probably impossible to open source as-is due to licensing agreements it still might not work; depends on how much of the plugin is still left after you've removed those bits.)