What kind of video servers are you talking about? For ingesting from a feed/camera in? Streaming out video? Curious.
My recent project, which was relatively small, played back a single video over 4 blended projectors. Each video frame was a 50MB uncompressed TGA file, 30 times a second. On a more complex show, you could be trying to play back multiple video streams simultaneously.
D3 - http://www.d3technologies.com/ and Pandora - http://www.coolux.de/products/pandoras-box-server/
are two of the big players in the industry.
Ouch! Not even a PNG?
I'm not an expert on the exact architecture, but I'm guessing that with uncompressed TGA you just throw the bits at the GPU and they get displayed, while if you have to uncompress images that first gets handled by the CPU (?).