> with tons of in-order cores

It's worth noting that GPUs are only "in order" because the computer architecture community has decided that "out of order" means "some variation of Tomasulo." Every GPU architecture in the last decade has relied extensively on executing (and committing) instructions in a single instruction stream out of order.

Doing a bunch of math while waiting for memory operations to complete is fundamental to high performance. Modern GPUs cannot achieve anywhere close to peak performance without extensive ILP - relying on latency hiding via threads is insufficient.