Kudos to the kids for asking first, and at first I didn't mind, but they ended up climbing on my pickup truck parked near the tree (the only place to park it) and dented the hood and roof in several places. After I showed them the damage and asked them not to climb on my truck, I had a gaggle of neighborhood moms beating my door down to tell me how much of a prick I was to forbid their children from climbing my truck to get to the tree. I ended up having to call the cops to have them removed.
When the landlord found out I called the cops on them, he took their side and asked me to find another place to live (I wasn't under a lease, it was a month to month rental agreement).
I'm a homeowner now, in a house that is not in a neighborhood specifically so I can avoid incidents where people feel they have a right to invade my space and damage my property. Again, I'd have no problem if I had a fruit tree in my yard today and people walking past wanted to get a bite to eat as long as they are courteous about it, but mob mentality is not something I'm comfortable with.