Lets go back in time to the modern days of the early to mid 90's. GPU's were very new. Back in that day if you wanted to interface with a GPU you had to know what language it spoke.
Every company had their own language, architecture, etc. So if you made software you had to make a different version for each hardware GPU. Case and point 5 version of MechWarrior, 1 version for each GPU company on the market.
:.:.:
Eventually we got Glide, Mantle, DirectX, and OpenGL. These aimed to standardize our interface to the GPU so that the GPU hardware guys can focus on building blindingly fast vector processors with no regard to architecture backwards compatibility, etc.
Want to completely change register layout every 2 years? No problem. Want to redo caching every 18 months? No problem.
This has lend to blindingly fast GPU scaling. And the market exploding as GPU's currently lead in GFLOPS/watt. Also allows for average customers to have access to TFLOPS level of computing which in turn have us the currently growing open science and data initiatives.
:.:.:
Pandora was let out of their box, its going to be very hard to put them back in. DirectX/OpenGL is more or less a Compiler for the byte code that LLVM/GCC generates for 3D acceleration. This byte code is then compiled into native code for the GPU based on the GPU driver for DirectX/OpenGL.
This is the problem. Your GPU isn't a device, so much a fully feature complete computer that is programmed on the fly by your CPU.