I saw Jack Dongarra say that plain parallelism is now old and the thing is “communication avoidance”. Also made the point (relevant to your quotation) that changing hardware means redoing the algorithms every ten years to exploit it.
Are there researchers working on finding the theoritical communicication limits of classical numerical problems?