And then you have illustrious hedge funds that go out of business when the SP500 suffers a small blip [1]. In any case, paying 2-5% to a hedge fund doesn't seem very good business when you're withdrawing 4% a year to live on.
[1] https://en.wikipedia.org/wiki/Long-Term_Capital_Management