I don't think Sun was particularly good at optimizing in this particular task --or even interested.
They worked towards their actual customer use, ie. the server side.
It's not just 3D code, even desktop UI libs (Swing, JavaFX) and multimedia code where left without much (or any) love from Sun (/Oracle).
That doesn't mean a VM is unsuitable for fast 3D. Don't tons of 3D games use the Unity engine, which utilizes the Mono VM?
And it's not like critical 3D parts of the implementation cannot be written in plain old C/C++/ASM -- and exposed to the web programmers via some VM interface (a special set of 3D specialized opcodes? an intermediary 3D extension language + lib other vm languages will have to use? I dunno, it's however totally possible)