If main memory is so slow, would it be better to treat cache as memory and handle it directly in code rather than try to trick it into doing what we want?
On the original PlayStation, the scratchpad was a PITA. You had to weigh the speedup against the cost of manually copying in and out the the pad. A modern implementation would copy via DMA like the PS3's Cell processors. Still a PITA, but at least the payoff for the effort can be quite good.
The other advantage of cache over explicit control of the on-chip SRAM is that it allows code to be inherently forwards and backwards compatible, rather than tying the code to a specific machine with a specific amount of on-chip memory. And I imagine the nightmare would only increase if you had to factor multitasking into an explicit scheme.
The primary benefit of explicit on-chip memory is not, AFAIK, that you can manually manage it significantly better than a cache, but that it takes up significantly less die space and has lower access latency. You really see this idea shine in tiny cheap microcontrollers that have no external memory.
It does not have to be coherent I think. Process thrashing sounds like an issue though. I imagine if the address space is across all CPU cores then the software can pin to a core so that the access is local. Remote access to another CPU will be slow, but I'm not sure how great it is with caches anyway.