The "just retry" approach is truly bothersome. I think it is at least partly an organizational issue, because it happens far more often when QA is a separate team.

I don't understand this at all. Why not just skip CI altogether if you're not interested in the results?

If a build fails 10% of the time, it actually takes 100x longer before to fail for the 10%x10%x10% case.