The ~4% SWR for FIRE assumes that we'll see sustained growth in equities markets, because that's what we've seen before. But the data really only goes back to the early 20th century (the data from the "Trinity Study" looked at the period from 1925 to 1995).
I agree with you that people in the FIRE camp will still be far better off than the average person. But to me the terrifying scenario is retiring at 30 and then running out of money at 60 because the 4% rule didn't work out.