I read "come to harm" as "get harmed" instead of "do harm".
In simpler terms: what's good for some is bad for others.
Yes, the idiom "come to X" does mean "be acted on by X". Common usages "come to harm", "come to Jesus", "come to his/her/your/my senses" ( = rational thought).
It's especially confusing in this case, where the previous sentence refers to a place.
"I come to the beach to relax, but others come to party."
“Don't it always seem to go
That you don't know what you've got
Till it's gone
They paved paradise
And put up a parking lot”