There can be overheads to running very large numbers of MPI processes, depending on how scalable the MPI implementation is, and there are overheads to having too many threads contend for memory on a single node. At least in the current state of things in HPC, there is a delicate balance to choosing the right process-thread ratio, and it differs from application to application and depends on the node count for the job. People struggle to find the sweet spot, but we're definitely not running all MPI anymore.
If you look in your HPC historybook, you'll notice that I was the system architect for InfiniPath.
A bit over-general, don't you think? There are many places that have big clusters full of single-threaded processes. There are also many places that have big clusters full of multi-threaded processes. I don't know which group is the biggest, but it doesn't matter as far as Osterhout's argument is concerned. What matters is that, contrary to the context in which that argument is framed, wringing performance out of multiple CPUs in a single system is an almost universal need nowadays. Many of them furthermore need high levels of intra-process as well as inter-process parallelism for that purpose. Threading is not the only way to achieve that, but he never even considers other alternatives. No matter what hairs are split or who tries to claim the biggest credentials, the OP remains seriously time-bound and its applicability to modern systems is weak.
(OK, in the guts, we did have some utility software that did async disk I/O using a few C threads per heavyweight process, but that was due to the limited kernel API more than us wanting threads.)
Glad to see HPC so represented here on HN, but an HPC counter-example does not diminish an example. To put this another way: I wasn't saying that YOUR problem could be solved without threads, I said MY problem could be solved without threads.
I was going to look up the current details for you, but I don't see an SGI Ultraviolet system in the current Top500. How large of a machine do you work with?