IIRC, the smallpox vaccine is the one that scars you for life. I bet a lot of people would be afraid of it at first, but a 30% fatality rate is probably high enough to change a lot of antivaxers' minds.
And anyway, if everyone has the scar we've all got the same tiny deduction in 'attractiveness,' so it cancels out I guess.
The risk of "inadvertent autoinoculation" was my bigger concern - if you touch the site before it's fully healed, you can spread the vaccina virus to other parts of your own body. In rare cases it's been known to cause blindness. Fortunately a Tegaderm dressing kept it contained to my shoulder, but damn did it itch like crazy for those few weeks.