A few examples of where this can go wrong:
* Introducing third party ballot collectors risks bad actors manipulating the ballot pipeline, for example what we saw in 2018 in North Carolina's 9th district.
* The US mail is mostly reliable, but lost mail (in my experience) is still fairly common. While this would be a random occurrence, every election cycle involves several tight races, and a lost ballot can flip the results.
* In the event errors in ballot transmission occur, you need to correct them in a timely manner. This requires someone to put in the effort to proactively watch out for missing ballots. Individuals are lazy, and it might not be in the interests of partisan officials to try.
The severity of these and other issues largely depends on the volume of mail voting; the more mail voting the more incentive to try to subvert it. As with any security issue, the defenses against them need to be strong enough to make the attacks too expensive to execute. Just because we don't see fraud yet doesn't mean mail voting won't be targeted in the future.