All the meaningful parts of the 3D engine (shader compiler, GPU code generator, etc...) are in userspace (in prebuilt binaries for MeeGo). The kernel doesn't do anything but take the output from those components and hand it to the hardware to execute.
I think I remember being told about a project to reverse engineer the SGX, but doubt it's made much progress. Even Nouveau is just barely usable for 3D now, and that's for much more accessible (and desirable) hardware.
Not necessarily. The idea behind clean room development is that one part of the team uses whatever legal techniques they want to analyse software, and document everything they discover. The documents are checked to make sure they don't contain any copyrightable information from the original (if information is not copyrightable, the methods used to obtain that information, be it decompiling or analysis of behaviour don't matter). A different team, who hasn't seen the original software at all, work off the document that doesn't contain any copyrightable information from the original.