I think the fear many people have is that digital ID will be required for non-government services as well. I can easily see that happen in the USA and Switzerland is the kind of weird that may also let that sort of thing happen.

With things like age verification becoming mandatory just about everywhere and actual privacy-conscious digital age verification being very difficult, there's definitely a risk towards abuse and badly designed authorization mechanisms (although the EU's open source backend and frontends should make it easy for other countries if they do actually care about privacy).