The raw throughputs are definitely better but there is concern regarding latency even at the lowest levels, since today's CPUs have complex cache, power state and frequency throttling mechanisms. You cannot guarantee that something will perform with identical runtime in all expected use-cases unless you take care to use hardware that is optimized in that direction. And because the software environments are more complex a lot of capability is just dropped on the floor because the intermediate layers get in the way.

W/R to buffers in front of things, a decent number of the micro systems of the 70's were so memory starved that they would make these tradeoffs to retain video sync - that describes the Atari 2600("racing the beam" is the name of a book about the console), and Cinematronics vector games(if it did not complete drawing at 60hz, it reset). Most early arcade games did work with DACs(or rather ADCs) but ran their own calibration and button debounce code - and even with layers of abstraction that's still basically true today.

With graphics, the move towards desktop environments doing GPU compositing impacts graphics coders, since they now often have window manager buffers in front of them instead of direct access to a rectangle of pixels.

Web browsers are a more extreme example of this. Because the graphics model revolves around document presentation, things that aren't really documents, or are extremely customized documents, often get burdened by latency that they wouldn't have if it weren't for the browser.