Most of the controversy is because they've never had a mandatory identification number before, and there's always going to be fear and resistance against introducing that kind of thing. I think Japan is a highly privacy-conscientious society, for instance it's still very cash-based.

The less populistic and more informed resistance is, as I see it, based on the fact that the My Number is designed to be a "secret" number you're only supposed to share with the authorities (as in, high risk of identity theft if the number is leaked) similar to the US SSN. But in order to try to increase popularity (and adoption), they're trying moves such as adding a grocery tax-refund scheme, which necessitates showing the ID card with the number prominently printed to minimum wage cashiers, etc. They've tried to placate fears with comments such as "but it's illegal for the cashiers to look at the number" and "you can have a new number issued 3 times!".

I'm from another country with a personal ID number. I find it very handy. But there it's not assumed to be secret (even some online shopping sites will accept it as entry to auto-populate your address from the population register), so there's some extra e.g. ID checks when you use it, lowering the risk of identity theft.