Not to defend the clearly overreaching actions of the Malibu homeowners, but hiring security guards to protect what you feel is your property is not vigalantism in any sense of the word.
A homeowner has every right to hire security guards for their property. At worst, the issue here is overreaching claims on the extent of their property – something that happens among even the unwealthy.