Being a "Malibu problem" doesn't really make this matter any less relevant. It's precisely this kind of being above the law feeling of the wealthy that's at the root of many much larger problems.
Not to defend the clearly overreaching actions of the Malibu homeowners, but hiring security guards to protect what you feel is your property is not vigalantism in any sense of the word.
A homeowner has every right to hire security guards for their property. At worst, the issue here is overreaching claims on the extent of their property – something that happens among even the unwealthy.
Hiring a guard for land you don't own is using illegal methods to break the law. (Assuming the guard actually does anything.)
I see this as worse than vigilantism, and I find it really odd to see you explain it as less bad. I seriously doubt that the majority of these landowners are actually mistaken about who owns the beach.
This isn't vigilantism. They're trying to mislead the public about their rights, not enforce some law with illegal methods. I'd be damned if any of our politicians were arrested for that, let alone some Malibu residents.
While the wealthy being above the law is a problem, the larger problem is the arbitrary nature of the law and the arbitrary nature of its enforcement. People wouldn't have to hire private guards if the law wasn't enforced. The police would enforce the law if there was less of it wasted on things that moral people do (like smoking pot).
Also reduced property taxes mean less police for everyone, so you get the "I only pay for what I need" anti-tax rationalization of those who live in gated and guarded neighborhoods which is ultimately socially destructive.