Interesting! This was exactly the case on the system I used. I didn't realize Cray was the only vendor who went the no-local-storage route.
The other problem is that their interconnect is relatively fragile. It's comparatively easy to crash the entire network, at which time your filesystem goes away and processing stops.
But thanks to Lustre, even when it's good, it's bad.