If processing latencies don't matter much, it's an easier more flexible system to use.
Because we have so much data (of the order of 25+ PB of raw data per year; it actually balloons to much more than this due to copies in many slightly different formats) and so many users (several thousand physicists on LHC experiments) that's why we have hundreds of GRID sites across the world. The scheduler sends your jobs to sites where the data is located. The output can then be transferred back via various academic/research internet networks.
HEP also tends to invent many of its own 'large-scale computing' solutions. For example most sites tend to use Condor[1] as the batch system, dcache[2] as the distributed storage system, XRootD[3] as the file access protocol, GridFTP[4] as the file transfer protocol. I know there are some sites that use Lustre but it's pretty uncommon.
[1] http://research.cs.wisc.edu/htcondor/ [2] http://www.dcache.org/ [3] http://xrootd.slac.stanford.edu/ [4] http://www.globus.org/toolkit/docs/latest-stable/gridftp/
Not for gauge configuration generation, which has to be done sequentially, but inversions and measurements can be mapped / reduced.
I do hope physics people consider MR systems in addition to the existing ones they use.
It is quite easy to implement map/reduce style programs in MPI. It is quite difficult or impossible to implement the parallel algorithms that win the Gordon Bell prize, almost all of which are implemented in MPI, using Hadoop.