In the US, it would be about ~50% of users, while in Europe it's ranging from 30% (France) to 98% (Spain) who wouldn't be able to visit the website.
But yeah, I'd do what you say in the bottom of your comment. Add AAAA records and then see how many people uses ipv6 compared to ipv4 and then decide.