Your manager can say "based on feedback everyone says you suck" and you can't really do a whole lot to look into it. You don't know where the feedback is coming from, or if it's even valid.
At a smaller company a manager seems more responsible to look at performance democratically. So if they have a personal vendetta and they fired someone for it, it's much easier for other employees and other managers to see what's going on and call it out (so they might not act on it to begin with).
Of course that's not universal, sometimes at a small company you might have one person with consolidated power so it doesn't really matter (and can be much worse). And of course, at a smaller company peer pressure can keep issues under wraps because you're worried about everyone knowing that you're the one bringing it up.
So yeah, a performance review process needs to exist at larger companies because it makes sense logistically... but criticism of that process doesn't mean it should be thrown out entirely, it just welcomes the discussion about how it can be improved. There are middle grounds between strict performance reviews and small-company democracy.