So long as some important applications have tons of similar threads and some have very few threads, it will probably make sense to specialize.
At some point, and I think 256 cores might be the ballpark, you're committing to using so many threads that you're probably mostly interested in high throughput. (I'm writing commercial path tracers so my bias is obvious!)
One speculates to hide memory latency, the other shuffles threads between cycles to hide memory latency. One has ~64 byte wide vector ops, one has ~256 byte wide vector ops.
I have a pet theory that the significant difference is the cache coherency model.
One of the uses they prototyped was a GPU, or a large multi-threaded (x86)software renderer rather than going through a regular 3D acceleration API. I remember reading that part of the challenge was that Larrabee was a system itself, so a developer needed to boot something like BSD before providing it with code to get useful output. This was around the time AMD was experimenting with 'fusion' after their purchase of AMD, and exploring how to push different different parts of an application to the relevant processor in their CPU+GPU products.
That's in addition to the other Xeon Phi accelerators they did. Obviously Sierra Forest is a regular CPU, but it seems there's a bit of "history doesn't repeat, but it rhymes"
Can't remember if it had a lower max core count across SKUs but at least one popular vtuber got hands on one.
Also, CPU cores tend to have speculative execution, GPUs don't.
So long as GPU threads have to run in a warp, they will be fundamentally very different from a CPU core, no matter how weak, strong, or power efficient.
Multi-core CPUs are for parallel processes. You want to host a ton of virtual machines and they all care about branch prediction and cache latency more than throughput.