It can definitely be an issue and a point of conflict in popular areas. It may be an issue that needs to be resolved with permits—limiting to a certain number of horse users per day per trailhead, for example. I hate permits, but if visitors are beginning to significantly impact the Wilderness (which the Wilderness Act is established to protect against) then permits to reduce impact are necessary.
Where I live, most Wilderness Areas are relatively little used and suffering from post-fire erosion, and so the trails often suffer from overgrowth and fading tread. As a result, horse use is actually quite welcome as it's currently below the threshold where it goes from helping keep the trail beaten in to actively damaging it.