On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel. Here’s a proof, I did it alone spending a month or so part time, and resource use is lower than VLC’s despite my code is in .NET: https://github.com/Const-me/Vrmac/tree/master/VrmacVideo
However, that wasn’t always the case. Until recently software had to fallback to software decoders, and implement other workarounds. Software developers are reluctant to throw away old code and do something completely different.
1. They spent much time on that old code (sunk cost fallacy).
2. Old code usually works more often than it doesn’t (don’t fix things which aren’t broken).
3. Massive refactor of large software is very expensive in general. Moving stuff from CPU to GPU is going to cause large changes in many parts. You can’t usually afford to copy uncompressed video frames from VRAM back to system RAM and call it a day, that would be too much bandwidth. Instead, you have to change a lot of other code to handle GPU textures instead of system RAM images.