If you mail people blank ballots, on the day the ballots go out someone could follow the mail truck around the neighborhood stealing them all out of the mailboxes, both disenfranchising everyone in that neighborhood and providing the theives with blank ballots they could fill out and submit.
It opens you up to fresh chicanery like giving people "updates" on where to mail their ballots and having voters send them to the perpetrators who then either modify the ballots before submitting them or only submit the favorable ones.
It creates the possibility that somebody figures out how to forge ballots and then prints off as many as they like and mails them in, because there are no poll workers there to observe anything.