He started backpeddling the minute people filled suit, as any true man of principle fighting the bad guys would do.
For years he made factual claims about specific people that are demonstrably and provably untrue (i.e. that they're "crisis actors" who faked the lot), which does not fall under Free Speech, and did cause damages because several of Jones' followers did harass various survivors and family members with their bullshit.
You should read one of the billion case summaries published this week. Maybe you'll see why "tons of people" have reacted the way they have.