The damage is to the serenity enjoyed by the people hiking or horseback riding, not the trail itself.
Also, remember that horses are not an indigenous species in recorded history in North America.
I don't think one should consider nativity very much when deciding if something is good or not in a given context. More what the impact is, irrespective of nativity.
Never the less, bikes are even less native than horses... At least horses are organic.