The "market average" like S&P is the expected return of passive investment - doing nothing, so that's why it's a reasonable benchmark. This (not 0%) is the straightforward default alternative investment, the one against you can measure an opportunity cost.
The point of hedge funds is that they can do better than that, but charge a fee. However, if you get worse results than doing nothing, then the fee seems unjustified.