Believe it or not, to be able to do that at a high throughput you need half of a GPU anyway, like a cache hierarchy.
This is a big part of why RISC didn't win and today the largest server chips in use in datacenters are still mostly compatible with an 8 bit part from the early 1980's.