Local participatory democracy is in fact pretty terrible: HOAs, school boards, neighborhood impact hearings where people complain that building apartments would let the poors move in and we can't have that.
I [given the chance] would also vote against apartments in my particular neighborhood, and it has nothing to do with being poor at all.
But it's not your fault, it's a systemic issue. The fact you even care about your property value is the problem, not the fact you have to care about it. That's not your fault, you're just playing the game you have to play like everyone else.
If you don't want to be on the defensive, try to make your position comprehensible.