I'd imagine the graphics libraries is where the JVM falls down.
Java history is full of OpenGL and DirectX bindings as well.
Just not all of them have survived the original authors leaving the project.
Before Unity was a thing, Java already had the Java Monkey Engine.
The biggest thing in java gamedev is avoiding GC pauses.
Languages like Clojure stress more the GC.