Our primary use case for Gluster is in serving persistent filesystems for Drupal. Our customers store potentially millions of files on their GlusterFS clusters.
We've built a number of tools/processes to help protect Gluster against failures in EC2 (for instance fencing network traffic at the iptables layer to help protect GlusterFS clients from hanging talking to down nodes), as well as to help our team perform common tasks (resizing clusters, moving customers from cluster to cluster, etc.). We haven't necessarily hit blocker issues recovering from underlying hardware fails, but our team is definitely very experienced with many possible failure modes.
Overall GlusterFS has been very reliable over the years and our research has shown it is the best option out there for when our customers can't use something such as S3 directly.
If you want more details or would love to hack on a 8000+ node EC2 cluster running things such as GlusterFS feel free to ping me.